[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:26 dist-test-slave-jvsj krb5kdc[13967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:26 dist-test-slave-jvsj krb5kdc[13967](info): set up 2 sockets
Sep 24 10:01:26 dist-test-slave-jvsj krb5kdc[13967](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 10:01:27.002585 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:27 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:27.163713 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36735
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:27.475415 13980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:27.475704 13980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:27.475826 13980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:27.483356 13980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:27.483492 13980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:27.483549 13980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:27.483633 13980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:27.483693 13980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:27.493770 13980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:27.495162 13980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:27.496472 13980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:27.508018 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:27.508081 13985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:27.508306 13980 server_base.cc:1034] running on GCE node
W20260924 10:01:27.508039 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:27.509445 13980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:27.510983 13980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:27.512194 13980 hybrid_clock.cc:648] HybridClock initialized: now 1790244087512142 us; error 43 us; skew 500 ppm
I20260924 10:01:27.512534 13980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:27 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244087, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:27.515514 13980 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:27.516957 13980 webserver.cc:460] Webserver started at http://127.13.162.62:37707/ using document root <none> and password file <none>
I20260924 10:01:27.517652 13980 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:27.517764 13980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:27.518041 13980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:27.520938 13980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "fcf3fd20fc80420a969f9c0f2d413d8b"
format_stamp: "Formatted at 2026-09-24 10:01:27 on dist-test-slave-jvsj"
server_key: "f1b618943a0fcdfa2fe2fe0b63b07975"
server_key_iv: "f781fb21b5082167a4ac20ee69024243"
server_key_version: "encryptionkey@0"
I20260924 10:01:27.521677 13980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "fcf3fd20fc80420a969f9c0f2d413d8b"
format_stamp: "Formatted at 2026-09-24 10:01:27 on dist-test-slave-jvsj"
server_key: "f1b618943a0fcdfa2fe2fe0b63b07975"
server_key_iv: "f781fb21b5082167a4ac20ee69024243"
server_key_version: "encryptionkey@0"
I20260924 10:01:27.526859 13980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:01:27.530243 13995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:27.531635 13980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:27.531793 13980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "fcf3fd20fc80420a969f9c0f2d413d8b"
format_stamp: "Formatted at 2026-09-24 10:01:27 on dist-test-slave-jvsj"
server_key: "f1b618943a0fcdfa2fe2fe0b63b07975"
server_key_iv: "f781fb21b5082167a4ac20ee69024243"
server_key_version: "encryptionkey@0"
I20260924 10:01:27.531965 13980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:27.556083 13980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:27.560456 13980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:27.560729 13980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:27.577402 13980 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:36735
I20260924 10:01:27.577425 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:36735 every 8 connection(s)
I20260924 10:01:27.578685 13980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:27.582469 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:27.589046 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 13980
I20260924 10:01:27.589366 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:27.591817 13960 external_mini_cluster.cc:1428] Setting key db9c32be1025e7d005c8d421499a535f
I20260924 10:01:27.593819 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b: Bootstrap starting.
I20260924 10:01:27.597455 14047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:27.598474 14047 log.cc:824] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:27.601193 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b: No bootstrap required, opened a new log
I20260924 10:01:27.610052 14047 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36735 } }
I20260924 10:01:27.610446 14047 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:27.610540 14047 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf3fd20fc80420a969f9c0f2d413d8b, State: Initialized, Role: FOLLOWER
I20260924 10:01:27.611101 14047 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36735 } }
I20260924 10:01:27.611317 14047 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:27.611442 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:27.611591 14047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:27.613801 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36735 } }
I20260924 10:01:27.614270 14047 leader_election.cc:302] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcf3fd20fc80420a969f9c0f2d413d8b; no voters:
I20260924 10:01:27.614902 14047 leader_election.cc:288] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:27.615034 14052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:27.616074 14052 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [term 1 LEADER]: Becoming Leader. State: Replica: fcf3fd20fc80420a969f9c0f2d413d8b, State: Running, Role: LEADER
I20260924 10:01:27.616626 14052 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36735 } }
I20260924 10:01:27.617115 14047 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:27.619328 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcf3fd20fc80420a969f9c0f2d413d8b. Latest consensus state: current_term: 1 leader_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36735 } } }
I20260924 10:01:27.619695 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:27.620227 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf3fd20fc80420a969f9c0f2d413d8b" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36735 } } }
I20260924 10:01:27.620496 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b [sys.catalog]: This master's current role is: LEADER
Sep 24 10:01:27 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:27.624341 14057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:27.630151 14057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:27.634135 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:27.596468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.13.162.62:36735)
0924 10:01:27.597399 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:27.597416 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:27.598704 (+ 1288us) server_negotiation.cc:407] Connection header received
0924 10:01:27.607708 (+ 9004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:27.607744 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:27.608320 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:27.608793 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:27.611755 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:27.613076 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:27.614820 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:27.615795 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:27.617002 (+ 1207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:27.622929 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:27.622942 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:27.622964 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:27.623027 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:27.626619 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:27.627634 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:27.627646 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:27.627653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:27.628078 (+ 425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:27.628657 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:27.628661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:27.628665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:27.629382 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:27.632056 (+ 2674us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:27.632347 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 10:01:27.632782 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":403,"threads_started":1}
I20260924 10:01:27.652700 14057 catalog_manager.cc:1347] Generated new cluster ID: 5702fb13b87e48d486a523d33d35ab7f
I20260924 10:01:27.652839 14057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:27.672652 14057 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:27.674562 14057 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:27.685705 14057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcf3fd20fc80420a969f9c0f2d413d8b: Generated new TSK 0
I20260924 10:01:27.686761 14057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:27.725310 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36735
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:28.033377 14073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:28.033680 14073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:28.033805 14073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:28.041291 14073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:28.041425 14073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:28.041580 14073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:28.051805 14073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36735
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:28.053259 14073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:28.054663 14073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:28.067478 14079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:28.067590 14081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:28.067521 14078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:28.068152 14073 server_base.cc:1034] running on GCE node
I20260924 10:01:28.068776 14073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:28.069687 14073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:28.070896 14073 hybrid_clock.cc:648] HybridClock initialized: now 1790244088070804 us; error 73 us; skew 500 ppm
I20260924 10:01:28.071206 14073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:28 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244088, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:28.074265 14073 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:28.075639 14073 webserver.cc:460] Webserver started at http://127.13.162.1:33983/ using document root <none> and password file <none>
I20260924 10:01:28.076298 14073 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:28.076421 14073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:28.076704 14073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:28.079629 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "c9cf5d5132a648d7a1ac04d84b2b72b0"
format_stamp: "Formatted at 2026-09-24 10:01:28 on dist-test-slave-jvsj"
server_key: "8af6b6829e0c72bf63bb0258329c6d53"
server_key_iv: "4abf42d5a972da63ff289802c99b0dc5"
server_key_version: "encryptionkey@0"
I20260924 10:01:28.080355 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "c9cf5d5132a648d7a1ac04d84b2b72b0"
format_stamp: "Formatted at 2026-09-24 10:01:28 on dist-test-slave-jvsj"
server_key: "8af6b6829e0c72bf63bb0258329c6d53"
server_key_iv: "4abf42d5a972da63ff289802c99b0dc5"
server_key_version: "encryptionkey@0"
I20260924 10:01:28.085510 14073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:01:28.088994 14088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:28.090484 14073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:28.090636 14073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "c9cf5d5132a648d7a1ac04d84b2b72b0"
format_stamp: "Formatted at 2026-09-24 10:01:28 on dist-test-slave-jvsj"
server_key: "8af6b6829e0c72bf63bb0258329c6d53"
server_key_iv: "4abf42d5a972da63ff289802c99b0dc5"
server_key_version: "encryptionkey@0"
I20260924 10:01:28.090808 14073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:28.117264 14073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:28.121627 14073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:28.121912 14073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:28.122911 14073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:28.124249 14073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:28.124325 14073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:28.124444 14073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:28.124498 14073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:28.153921 14073 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:39619
I20260924 10:01:28.153944 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:39619 every 8 connection(s)
I20260924 10:01:28.155242 14073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:28.161135 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14073
I20260924 10:01:28.161363 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:28.162132 13960 external_mini_cluster.cc:1428] Setting key a0dc9ca8b42658954991287218b64779
Sep 24 10:01:28 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244088, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:28.176318 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:28.157690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52659 (local address 127.13.162.62:36735)
0924 10:01:28.158110 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:28.158114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:28.159132 (+ 1018us) server_negotiation.cc:407] Connection header received
0924 10:01:28.160999 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:28.161003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:28.161074 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:28.161169 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:28.163141 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.164158 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.166556 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.166935 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.167038 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:28.170745 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:28.170750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:28.170753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:28.170784 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:28.173154 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:28.174006 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:28.174010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:28.174012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:28.174129 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:28.174603 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:28.174606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:28.174608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:28.174860 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:28.174964 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:28.175898 (+ 934us) server_negotiation.cc:299] Negotiation successful
0924 10:01:28.176060 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":203,"threads_started":1}
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:01:28.177815 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:28.158255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36735 (local address 127.13.162.1:52659)
0924 10:01:28.158980 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:28.159020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:28.160676 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:28.161439 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:28.161451 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:28.162178 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:28.162954 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:28.162981 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.164345 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.164350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:28.166368 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:28.166389 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.167142 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.167149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:28.167410 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:28.168076 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:28.168104 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:28.170508 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:28.173282 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:28.173292 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:28.173309 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:28.173877 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:28.174230 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:28.174234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:28.174238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:28.174505 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:28.174971 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:28.174981 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:28.175405 (+ 424us) client_negotiation.cc:769] Sending connection context
0924 10:01:28.175776 (+ 371us) client_negotiation.cc:240] Negotiation successful
0924 10:01:28.176333 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":247,"threads_started":1}
I20260924 10:01:28.179452 14201 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36735
I20260924 10:01:28.179778 14201 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:01:28.180595 14201 heartbeater.cc:511] Master 127.13.162.62:36735 requested a full tablet report, sending...
I20260924 10:01:28.183200 14012 ts_manager.cc:194] Registered new tserver with Master: c9cf5d5132a648d7a1ac04d84b2b72b0 (127.13.162.1:39619)
I20260924 10:01:28.185006 14012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:52659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:28.199481 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36735
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:28.521209 14207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:28.521498 14207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:28.521592 14207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:28.528872 14207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:28.528995 14207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:28.529110 14207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:28.538998 14207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36735
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:28.540249 14207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:28.541524 14207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:28.554168 14212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:28.554204 14215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:28.554237 14213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:28.555627 14207 server_base.cc:1034] running on GCE node
I20260924 10:01:28.556227 14207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:28.557178 14207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:28.558408 14207 hybrid_clock.cc:648] HybridClock initialized: now 1790244088558336 us; error 58 us; skew 500 ppm
I20260924 10:01:28.558727 14207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:28 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244088, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:28.565896 14207 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:28.567504 14207 webserver.cc:460] Webserver started at http://127.13.162.2:36741/ using document root <none> and password file <none>
I20260924 10:01:28.568215 14207 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:28.568331 14207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:28.568626 14207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:28.571588 14207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "3540cdde8a7e40c58e3bb76dfc1937c1"
format_stamp: "Formatted at 2026-09-24 10:01:28 on dist-test-slave-jvsj"
server_key: "645d2e6a28d089e35539566b03058d95"
server_key_iv: "d31afc5c08bf6a4a46e847d19e4fde82"
server_key_version: "encryptionkey@0"
I20260924 10:01:28.572360 14207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "3540cdde8a7e40c58e3bb76dfc1937c1"
format_stamp: "Formatted at 2026-09-24 10:01:28 on dist-test-slave-jvsj"
server_key: "645d2e6a28d089e35539566b03058d95"
server_key_iv: "d31afc5c08bf6a4a46e847d19e4fde82"
server_key_version: "encryptionkey@0"
I20260924 10:01:28.577580 14207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:01:28.584749 14222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:28.586428 14207 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20260924 10:01:28.586676 14207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "3540cdde8a7e40c58e3bb76dfc1937c1"
format_stamp: "Formatted at 2026-09-24 10:01:28 on dist-test-slave-jvsj"
server_key: "645d2e6a28d089e35539566b03058d95"
server_key_iv: "d31afc5c08bf6a4a46e847d19e4fde82"
server_key_version: "encryptionkey@0"
I20260924 10:01:28.586870 14207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:28.614567 14207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:28.618923 14207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:28.619184 14207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:28.620137 14207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:28.621454 14207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:28.621529 14207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:28.621644 14207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:28.621694 14207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:28.650571 14207 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:36303
I20260924 10:01:28.650620 14334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:36303 every 8 connection(s)
I20260924 10:01:28.651811 14207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:28.655781 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14207
I20260924 10:01:28.656003 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:28.656477 13960 external_mini_cluster.cc:1428] Setting key 4e77044002faa3c97f137c41292fa7bf
Sep 24 10:01:28 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244088, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:01:28.672883 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:28.654400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:55007 (local address 127.13.162.62:36735)
0924 10:01:28.654554 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:28.654559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:28.656008 (+ 1449us) server_negotiation.cc:407] Connection header received
0924 10:01:28.658289 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:28.658294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:28.658408 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:28.658503 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:28.659998 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.660692 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.662487 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.662914 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.663017 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:28.666805 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:28.666812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:28.666816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:28.666855 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:28.669880 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:28.670646 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:28.670651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:28.670655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:28.670808 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:28.671287 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:28.671292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:28.671295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:28.671623 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:28.671734 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:28.672460 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 10:01:28.672652 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:01:28.673961 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:28.654980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36735 (local address 127.13.162.2:55007)
0924 10:01:28.655813 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:28.655857 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:28.658011 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:28.658747 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:28.658755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:28.659272 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:28.659842 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:28.659860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.660830 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.660836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:28.662306 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:28.662349 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:28.663124 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:28.663130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:28.663389 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:28.664099 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:28.664125 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:28.666592 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:28.670024 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:28.670031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:28.670045 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:28.670519 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:28.670943 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:28.670946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:28.670949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:28.671169 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:28.671742 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:28.671748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:28.672127 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 10:01:28.672368 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 10:01:28.672763 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":297,"threads_started":1}
I20260924 10:01:28.675449 14335 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36735
I20260924 10:01:28.675786 14335 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:28.676584 14335 heartbeater.cc:511] Master 127.13.162.62:36735 requested a full tablet report, sending...
I20260924 10:01:28.678078 14012 ts_manager.cc:194] Registered new tserver with Master: 3540cdde8a7e40c58e3bb76dfc1937c1 (127.13.162.2:36303)
I20260924 10:01:28.678968 14012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:55007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:28.691214 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36735
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:29.000123 14340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:29.000391 14340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:29.000483 14340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:29.007611 14340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:29.007722 14340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:29.007834 14340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:29.017570 14340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36735
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:29.018919 14340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:29.020200 14340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:29.032564 14348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:29.032601 14346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:29.032621 14345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:29.033565 14340 server_base.cc:1034] running on GCE node
I20260924 10:01:29.034200 14340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:29.035223 14340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:29.036441 14340 hybrid_clock.cc:648] HybridClock initialized: now 1790244089036371 us; error 58 us; skew 500 ppm
I20260924 10:01:29.036751 14340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:29.039857 14340 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:29.041292 14340 webserver.cc:460] Webserver started at http://127.13.162.3:42053/ using document root <none> and password file <none>
I20260924 10:01:29.041957 14340 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:29.042053 14340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:29.042346 14340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:29.045275 14340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "37ff88ad71774ab890261c8e394ba199"
format_stamp: "Formatted at 2026-09-24 10:01:29 on dist-test-slave-jvsj"
server_key: "090cf31d46d81cc3e2cc3c8f14d39f67"
server_key_iv: "52e7c33d2d12f14c16f6d41ddc8f6a11"
server_key_version: "encryptionkey@0"
I20260924 10:01:29.045974 14340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "37ff88ad71774ab890261c8e394ba199"
format_stamp: "Formatted at 2026-09-24 10:01:29 on dist-test-slave-jvsj"
server_key: "090cf31d46d81cc3e2cc3c8f14d39f67"
server_key_iv: "52e7c33d2d12f14c16f6d41ddc8f6a11"
server_key_version: "encryptionkey@0"
I20260924 10:01:29.050948 14340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 10:01:29.054479 14355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:29.055935 14340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:29.056090 14340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "37ff88ad71774ab890261c8e394ba199"
format_stamp: "Formatted at 2026-09-24 10:01:29 on dist-test-slave-jvsj"
server_key: "090cf31d46d81cc3e2cc3c8f14d39f67"
server_key_iv: "52e7c33d2d12f14c16f6d41ddc8f6a11"
server_key_version: "encryptionkey@0"
I20260924 10:01:29.056263 14340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:29.069200 14340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:29.073242 14340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:29.073503 14340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:29.074451 14340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:29.075745 14340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:29.075814 14340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:29.075937 14340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:29.075986 14340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:29.104892 14340 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:40257
I20260924 10:01:29.104924 14467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:40257 every 8 connection(s)
I20260924 10:01:29.106235 14340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:29.116169 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14340
I20260924 10:01:29.116356 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:29.116904 13960 external_mini_cluster.cc:1428] Setting key 2326d9376cf236e9c8e616a53ef9b54d
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:29.125562 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.108651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:54435 (local address 127.13.162.62:36735)
0924 10:01:29.108855 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:29.108861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:29.110056 (+ 1195us) server_negotiation.cc:407] Connection header received
0924 10:01:29.111850 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:29.111854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:29.111927 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:29.112020 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:29.113491 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.114236 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.115753 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.116153 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.116252 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.120361 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:29.120366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:29.120370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:29.120405 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:29.122994 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.123654 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.123658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.123661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.123778 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.124145 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.124149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.124151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.124416 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:29.124520 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:29.125150 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 10:01:29.125310 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:01:29.126577 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.109221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36735 (local address 127.13.162.3:54435)
0924 10:01:29.109897 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:29.109939 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:29.111567 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:29.112217 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:29.112225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:29.112748 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:29.113310 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:29.113326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.114385 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.114388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:29.115629 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:29.115641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.116375 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.116382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:29.116684 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.117422 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:29.117444 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:29.120126 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:29.123124 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:29.123131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:29.123144 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:29.123546 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:29.123853 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:29.123856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:29.123858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:29.124057 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:29.124509 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:29.124515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:29.124828 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 10:01:29.125070 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 10:01:29.125445 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260924 10:01:29.127887 14468 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36735
I20260924 10:01:29.128154 14468 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:29.128803 14468 heartbeater.cc:511] Master 127.13.162.62:36735 requested a full tablet report, sending...
I20260924 10:01:29.130070 14012 ts_manager.cc:194] Registered new tserver with Master: 37ff88ad71774ab890261c8e394ba199 (127.13.162.3:40257)
I20260924 10:01:29.130865 14012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:54435
I20260924 10:01:29.133037 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:29.173069 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.158850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.13.162.1:39619)
0924 10:01:29.159536 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:29.159543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:29.159571 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:01:29.159720 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:29.159727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:29.159999 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:29.160128 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:29.161539 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.162580 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.163759 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.164345 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.164454 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.168033 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:29.168044 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:29.168057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:29.168091 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:29.170599 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.171156 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.171162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.171164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.171276 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.171612 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.171616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.171619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.172042 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:29.172278 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:29.172497 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 10:01:29.172687 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":382,"threads_started":1}
W20260924 10:01:29.174245 14116 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55122
I20260924 10:01:29.187986 14201 heartbeater.cc:503] Master 127.13.162.62:36735 was elected leader, sending a full tablet report...
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:29.212250 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.199547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.13.162.1:39619)
0924 10:01:29.199733 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:29.199740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:29.200079 (+ 339us) server_negotiation.cc:407] Connection header received
0924 10:01:29.200245 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:29.200249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:29.200337 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:29.200417 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:29.201667 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.202368 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.203528 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.203928 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.204039 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.207594 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:29.207599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:29.207602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:29.207634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:29.209924 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.210557 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.210562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.210564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.210679 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.211011 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.211016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.211020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.211261 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:29.211469 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:29.211749 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 10:01:29.211949 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:01:29.213554 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14073
I20260924 10:01:29.225421 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.214492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.13.162.1:39619)
0924 10:01:29.214675 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:29.214680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:29.214846 (+ 166us) server_negotiation.cc:407] Connection header received
0924 10:01:29.214955 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:29.214959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:29.215017 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:29.215091 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:29.216118 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.216767 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.218028 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.218535 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.218631 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.220518 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:29.220522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:29.220525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:29.220552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:29.222923 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.223515 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.223521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.223524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.223646 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.224057 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.224061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.224064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.224358 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:29.224577 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:29.224840 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 10:01:29.225071 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:01:29.226761 14196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:29.308139 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14073
I20260924 10:01:29.320955 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14207
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:01:29.336093 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.321818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.13.162.2:36303)
0924 10:01:29.322338 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:29.322349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:29.322385 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:01:29.322443 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:29.322449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:29.322668 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:29.322790 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:29.323917 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.324896 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.326089 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.326708 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.326821 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.330767 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:29.330778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:29.330793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:29.330834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:29.333462 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.334045 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.334052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.334054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.334175 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.334517 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.334520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.334522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.334844 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:29.335140 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:29.335478 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 10:01:29.335724 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":261,"threads_started":1}
I20260924 10:01:29.337266 14330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:29.417120 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14207
I20260924 10:01:29.429136 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14340
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[13967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:01:29.444298 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:29.430114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.13.162.3:40257)
0924 10:01:29.430681 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:29.430688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:29.430716 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:01:29.430772 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:29.430778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:29.431074 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:29.431239 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:29.432651 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.433645 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.434925 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:29.435487 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:29.435576 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:29.439458 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:29.439471 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:29.439485 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:29.439515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:29.441936 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.442581 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.442587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.442590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.442702 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:29.442998 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:29.443001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:29.443003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:29.443299 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:29.443507 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:29.443819 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 10:01:29.444027 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":298,"threads_started":1}
I20260924 10:01:29.445590 14463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:29.527534 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14340
I20260924 10:01:29.539881 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 13980
I20260924 10:01:29.541054 14042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:29.639510 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 13980
2026-09-24T10:01:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3639 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[14489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[14489](info): set up 2 sockets
Sep 24 10:01:29 dist-test-slave-jvsj krb5kdc[14489](info): commencing operation
krb5kdc: starting...
W20260924 10:01:30.617936 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:30 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:30.776160 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:43015
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:43015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:31.078708 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:31.079015 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:31.079144 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:31.086244 14502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:31.086396 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:31.086478 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:31.086558 14502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:31.086617 14502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:31.096256 14502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:43015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:43015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:31.097565 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:31.098850 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:31.109794 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:31.109885 14510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:31.109929 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:31.110829 14502 server_base.cc:1034] running on GCE node
I20260924 10:01:31.111589 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:31.113045 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:31.114259 14502 hybrid_clock.cc:648] HybridClock initialized: now 1790244091114201 us; error 48 us; skew 500 ppm
I20260924 10:01:31.114665 14502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:31 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244091, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:31.117754 14502 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:31.119184 14502 webserver.cc:460] Webserver started at http://127.13.162.62:39471/ using document root <none> and password file <none>
I20260924 10:01:31.119834 14502 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:31.119932 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:31.120222 14502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:31.123212 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "16500e9f833e40f18c08edd7c6476e8f"
format_stamp: "Formatted at 2026-09-24 10:01:31 on dist-test-slave-jvsj"
server_key: "bbc4f8226154a1f9fd3f30dbcf8c365d"
server_key_iv: "33d9fdeb26a45616be11322989b0f638"
server_key_version: "encryptionkey@0"
I20260924 10:01:31.123937 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "16500e9f833e40f18c08edd7c6476e8f"
format_stamp: "Formatted at 2026-09-24 10:01:31 on dist-test-slave-jvsj"
server_key: "bbc4f8226154a1f9fd3f30dbcf8c365d"
server_key_iv: "33d9fdeb26a45616be11322989b0f638"
server_key_version: "encryptionkey@0"
I20260924 10:01:31.129055 14502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:01:31.132580 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:31.133960 14502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:31.134101 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "16500e9f833e40f18c08edd7c6476e8f"
format_stamp: "Formatted at 2026-09-24 10:01:31 on dist-test-slave-jvsj"
server_key: "bbc4f8226154a1f9fd3f30dbcf8c365d"
server_key_iv: "33d9fdeb26a45616be11322989b0f638"
server_key_version: "encryptionkey@0"
I20260924 10:01:31.134275 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:31.159011 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:31.163144 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:31.163368 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:31.179975 14502 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:43015
I20260924 10:01:31.179982 14568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:43015 every 8 connection(s)
I20260924 10:01:31.181239 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:31.184926 14569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:31.191005 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14502
I20260924 10:01:31.191220 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:31.191808 13960 external_mini_cluster.cc:1428] Setting key 91eed2084b7e8bd3d7151af1e5a61c77
I20260924 10:01:31.195335 14569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f: Bootstrap starting.
I20260924 10:01:31.198839 14569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:31.199890 14569 log.cc:824] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:31.202729 14569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f: No bootstrap required, opened a new log
Sep 24 10:01:31 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:31.211416 14569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16500e9f833e40f18c08edd7c6476e8f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43015 } }
I20260924 10:01:31.211755 14569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:31.211891 14569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16500e9f833e40f18c08edd7c6476e8f, State: Initialized, Role: FOLLOWER
I20260924 10:01:31.212492 14569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16500e9f833e40f18c08edd7c6476e8f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43015 } }
I20260924 10:01:31.212762 14569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:31.212958 14569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:31.213125 14569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:31.214003 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:31.193806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.13.162.62:43015)
0924 10:01:31.194591 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:31.194603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:31.194642 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:01:31.196290 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:31.196314 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:31.196764 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:31.197141 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:31.198894 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:31.200219 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:31.201678 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:31.202358 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:31.202494 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:31.206856 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:31.206865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:31.206885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:31.206927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:31.209833 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:31.210622 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:31.210629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:31.210635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:31.210787 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:31.211185 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:31.211189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:31.211191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:31.211703 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:31.211988 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:31.212485 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 10:01:31.212907 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":326,"threads_started":1}
I20260924 10:01:31.215502 14569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16500e9f833e40f18c08edd7c6476e8f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43015 } }
I20260924 10:01:31.215981 14569 leader_election.cc:302] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16500e9f833e40f18c08edd7c6476e8f; no voters:
I20260924 10:01:31.216576 14569 leader_election.cc:288] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:31.216660 14574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:31.217358 14574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [term 1 LEADER]: Becoming Leader. State: Replica: 16500e9f833e40f18c08edd7c6476e8f, State: Running, Role: LEADER
I20260924 10:01:31.217857 14574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16500e9f833e40f18c08edd7c6476e8f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43015 } }
I20260924 10:01:31.218566 14569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:31.220474 14575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16500e9f833e40f18c08edd7c6476e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16500e9f833e40f18c08edd7c6476e8f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43015 } } }
I20260924 10:01:31.220934 14575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:31.221307 14576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16500e9f833e40f18c08edd7c6476e8f. Latest consensus state: current_term: 1 leader_uuid: "16500e9f833e40f18c08edd7c6476e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16500e9f833e40f18c08edd7c6476e8f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43015 } } }
I20260924 10:01:31.221628 14576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:31.223031 14579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:31.228003 14579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:31.237581 14579 catalog_manager.cc:1347] Generated new cluster ID: 5372fef5d8774e488f53f0e1862a9e66
I20260924 10:01:31.237703 14579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:31.251956 14579 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:31.253276 14579 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:31.260766 14579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16500e9f833e40f18c08edd7c6476e8f: Generated new TSK 0
I20260924 10:01:31.261754 14579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:31.306381 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43015
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:31.615314 14595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:31.615590 14595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:31.615715 14595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:31.622938 14595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:31.623049 14595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:31.623207 14595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:31.635222 14595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:31.636497 14595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:31.637905 14595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:31.650764 14601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:31.650815 14600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:31.650933 14603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:31.651640 14595 server_base.cc:1034] running on GCE node
I20260924 10:01:31.652240 14595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:31.653200 14595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:31.654402 14595 hybrid_clock.cc:648] HybridClock initialized: now 1790244091654355 us; error 37 us; skew 500 ppm
I20260924 10:01:31.654707 14595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:31 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244091, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:31.657581 14595 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:31.659041 14595 webserver.cc:460] Webserver started at http://127.13.162.1:38165/ using document root <none> and password file <none>
I20260924 10:01:31.659706 14595 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:31.659818 14595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:31.660109 14595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:31.663090 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "6c12eab1bc04410aa38bd9f45085b0f5"
format_stamp: "Formatted at 2026-09-24 10:01:31 on dist-test-slave-jvsj"
server_key: "b750d45090ccba0f9ca430ea3de66332"
server_key_iv: "e0f736ecc75701cb6236b97858ef19ef"
server_key_version: "encryptionkey@0"
I20260924 10:01:31.663851 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "6c12eab1bc04410aa38bd9f45085b0f5"
format_stamp: "Formatted at 2026-09-24 10:01:31 on dist-test-slave-jvsj"
server_key: "b750d45090ccba0f9ca430ea3de66332"
server_key_iv: "e0f736ecc75701cb6236b97858ef19ef"
server_key_version: "encryptionkey@0"
I20260924 10:01:31.668757 14595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:01:31.672147 14610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:31.673413 14595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:01:31.673568 14595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "6c12eab1bc04410aa38bd9f45085b0f5"
format_stamp: "Formatted at 2026-09-24 10:01:31 on dist-test-slave-jvsj"
server_key: "b750d45090ccba0f9ca430ea3de66332"
server_key_iv: "e0f736ecc75701cb6236b97858ef19ef"
server_key_version: "encryptionkey@0"
I20260924 10:01:31.673748 14595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:31.686111 14595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:31.689986 14595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:31.690241 14595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:31.691150 14595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:31.692394 14595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:31.692477 14595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:31.692592 14595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:31.692646 14595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:31.721750 14595 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41461
I20260924 10:01:31.721809 14722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41461 every 8 connection(s)
I20260924 10:01:31.723078 14595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:31.732575 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14595
I20260924 10:01:31.732788 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:31.733347 13960 external_mini_cluster.cc:1428] Setting key 9d7afe7abae69025b68e1ac017cc4918
Sep 24 10:01:31 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244091, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:31.742003 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:31.725684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:41925 (local address 127.13.162.62:43015)
0924 10:01:31.726105 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:31.726112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:31.727113 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 10:01:31.728808 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:31.728812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:31.728879 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:31.728976 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:31.730374 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:31.731062 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:31.732581 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:31.733033 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:31.733142 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:31.736426 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:31.736431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:31.736434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:31.736464 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:31.739047 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:31.739889 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:31.739894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:31.739897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:31.740030 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:31.740389 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:31.740394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:31.740397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:31.740721 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:31.740854 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:31.741462 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 10:01:31.741668 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260924 10:01:31.742945 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:31.726272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43015 (local address 127.13.162.1:41925)
0924 10:01:31.726954 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:31.727004 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:31.728577 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:31.729178 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:31.729186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:31.729655 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:31.730203 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:31.730220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:31.731189 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:31.731192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:31.732437 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:31.732450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:31.733249 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:31.733256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:31.733487 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:31.734166 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:31.734187 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:31.736223 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:31.739187 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:31.739196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:31.739213 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:31.739778 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:31.740126 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:31.740129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:31.740132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:31.740302 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:31.740820 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:31.740825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:31.741137 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 10:01:31.741354 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 10:01:31.741739 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":247,"threads_started":1}
I20260924 10:01:31.744328 14723 heartbeater.cc:347] Connected to a master server at 127.13.162.62:43015
I20260924 10:01:31.744611 14723 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:31.745292 14723 heartbeater.cc:511] Master 127.13.162.62:43015 requested a full tablet report, sending...
I20260924 10:01:31.747259 14534 ts_manager.cc:194] Registered new tserver with Master: 6c12eab1bc04410aa38bd9f45085b0f5 (127.13.162.1:41461)
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:01:31.748698 14534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:41925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:31.767266 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43015
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:32.070839 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:32.071110 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:32.071231 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:32.078259 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:32.078384 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:32.078533 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:32.088028 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:32.089237 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:32.090579 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:32.103174 14734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:32.103245 14737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:32.103507 14729 server_base.cc:1034] running on GCE node
W20260924 10:01:32.103199 14735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:32.104337 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:32.105225 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:32.106411 14729 hybrid_clock.cc:648] HybridClock initialized: now 1790244092106359 us; error 35 us; skew 500 ppm
I20260924 10:01:32.106719 14729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:32 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244092, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:32.109540 14729 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:32.110973 14729 webserver.cc:460] Webserver started at http://127.13.162.2:40547/ using document root <none> and password file <none>
I20260924 10:01:32.111614 14729 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:32.111706 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:32.111956 14729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:32.114773 14729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91"
format_stamp: "Formatted at 2026-09-24 10:01:32 on dist-test-slave-jvsj"
server_key: "a456f263450cffa996b7db3afa0e01b6"
server_key_iv: "3c050bad1ee739fa7cfbd34ab522c04e"
server_key_version: "encryptionkey@0"
I20260924 10:01:32.115491 14729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91"
format_stamp: "Formatted at 2026-09-24 10:01:32 on dist-test-slave-jvsj"
server_key: "a456f263450cffa996b7db3afa0e01b6"
server_key_iv: "3c050bad1ee739fa7cfbd34ab522c04e"
server_key_version: "encryptionkey@0"
I20260924 10:01:32.120543 14729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:01:32.123945 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:32.125312 14729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:01:32.125471 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91"
format_stamp: "Formatted at 2026-09-24 10:01:32 on dist-test-slave-jvsj"
server_key: "a456f263450cffa996b7db3afa0e01b6"
server_key_iv: "3c050bad1ee739fa7cfbd34ab522c04e"
server_key_version: "encryptionkey@0"
I20260924 10:01:32.125645 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:32.142918 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:32.146662 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:32.146912 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:32.147856 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:32.149116 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:32.149188 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:32.149300 14729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:32.149361 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:32.177732 14729 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:44255
I20260924 10:01:32.177789 14856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:44255 every 8 connection(s)
I20260924 10:01:32.179016 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:32.182144 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14729
I20260924 10:01:32.182343 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:32.182828 13960 external_mini_cluster.cc:1428] Setting key 8e7cd8496f26d583bc9df110d0242b9c
Sep 24 10:01:32 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244092, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:01:32.200006 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.181653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:55969 (local address 127.13.162.62:43015)
0924 10:01:32.181817 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.181876 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.183474 (+ 1598us) server_negotiation.cc:407] Connection header received
0924 10:01:32.185563 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.185569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.185661 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.185757 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:32.187791 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.188897 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.190551 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.190965 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.191069 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.194356 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:32.194364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:32.194369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:32.194405 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:32.197023 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:32.197761 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:32.197765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:32.197768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:32.197906 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:32.198299 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:32.198303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:32.198305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:32.198594 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:32.198720 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.199383 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.199598 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:01:32.200933 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.182270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43015 (local address 127.13.162.2:55969)
0924 10:01:32.183306 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.183363 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.185264 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.186030 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.186041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.186786 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:32.187598 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.187625 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.189064 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.189070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.190421 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.190433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.191179 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.191186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.191423 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.192044 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:32.192070 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:32.194116 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:32.197158 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:32.197167 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:32.197185 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:32.197636 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:32.198008 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:32.198012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:32.198016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:32.198192 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:32.198711 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:32.198719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:32.199053 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.199279 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.199796 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":314,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:01:32.202363 14857 heartbeater.cc:347] Connected to a master server at 127.13.162.62:43015
I20260924 10:01:32.202687 14857 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:32.203449 14857 heartbeater.cc:511] Master 127.13.162.62:43015 requested a full tablet report, sending...
I20260924 10:01:32.204891 14534 ts_manager.cc:194] Registered new tserver with Master: 3ac3cd624d3b48d9bd09d8ede4edcf91 (127.13.162.2:44255)
I20260924 10:01:32.205950 14534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:55969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:32.221446 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43015
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:32.528162 14862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:32.528445 14862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:32.528548 14862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:32.535625 14862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:32.535727 14862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:32.535842 14862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:32.545318 14862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:32.546536 14862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:32.547760 14862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:32.560241 14870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:32.560268 14868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:32.560539 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:32.560976 14862 server_base.cc:1034] running on GCE node
I20260924 10:01:32.561578 14862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:32.562547 14862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:32.563753 14862 hybrid_clock.cc:648] HybridClock initialized: now 1790244092563678 us; error 63 us; skew 500 ppm
I20260924 10:01:32.564123 14862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:32 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244092, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:32.567174 14862 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:32.568601 14862 webserver.cc:460] Webserver started at http://127.13.162.3:41381/ using document root <none> and password file <none>
I20260924 10:01:32.569312 14862 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:32.569423 14862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:32.569698 14862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:32.572551 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "c29726605e0949d3a9cd4da3efa0d388"
format_stamp: "Formatted at 2026-09-24 10:01:32 on dist-test-slave-jvsj"
server_key: "b9eede08fcf691f05b232937c8e855c6"
server_key_iv: "7ce3113345364a6919fb56a267a6c033"
server_key_version: "encryptionkey@0"
I20260924 10:01:32.573278 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "c29726605e0949d3a9cd4da3efa0d388"
format_stamp: "Formatted at 2026-09-24 10:01:32 on dist-test-slave-jvsj"
server_key: "b9eede08fcf691f05b232937c8e855c6"
server_key_iv: "7ce3113345364a6919fb56a267a6c033"
server_key_version: "encryptionkey@0"
I20260924 10:01:32.578284 14862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:01:32.581655 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:32.583000 14862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:32.583141 14862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "c29726605e0949d3a9cd4da3efa0d388"
format_stamp: "Formatted at 2026-09-24 10:01:32 on dist-test-slave-jvsj"
server_key: "b9eede08fcf691f05b232937c8e855c6"
server_key_iv: "7ce3113345364a6919fb56a267a6c033"
server_key_version: "encryptionkey@0"
I20260924 10:01:32.583317 14862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:32.594801 14862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:32.598680 14862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:32.598937 14862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:32.599817 14862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:32.601090 14862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:32.601162 14862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:32.601272 14862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:32.601321 14862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:32.630921 14862 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:37939
I20260924 10:01:32.630990 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:37939 every 8 connection(s)
I20260924 10:01:32.632143 14862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:32.636073 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 14862
I20260924 10:01:32.636272 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:32.636822 13960 external_mini_cluster.cc:1428] Setting key 93c4f422d6dcbbda7109031de2c27fec
Sep 24 10:01:32 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244092, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:32.651829 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.634535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46423 (local address 127.13.162.62:43015)
0924 10:01:32.634697 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.634702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.635944 (+ 1242us) server_negotiation.cc:407] Connection header received
0924 10:01:32.638276 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.638280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.638383 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.638464 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:32.640162 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.640867 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.642403 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.642802 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.642889 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.646286 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:32.646291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:32.646295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:32.646358 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:32.648708 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:32.649484 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:32.649488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:32.649490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:32.649593 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:32.650004 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:32.650008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:32.650010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:32.650366 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:32.650491 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.651290 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.651496 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:01:32.652907 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.635029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43015 (local address 127.13.162.3:46423)
0924 10:01:32.635716 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.635767 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.638000 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.638767 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.638791 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.639316 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:32.640019 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.640040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.641013 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.641016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.642257 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.642273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.643097 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.643102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.643319 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.643968 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:32.643990 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:32.646086 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:32.648856 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:32.648862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:32.648876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:32.649382 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:32.649685 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:32.649688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:32.649690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:32.649915 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:32.650491 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:32.650499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:32.650906 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.651205 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.651739 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":253,"threads_started":1}
I20260924 10:01:32.654206 14990 heartbeater.cc:347] Connected to a master server at 127.13.162.62:43015
I20260924 10:01:32.654565 14990 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:32.655249 14990 heartbeater.cc:511] Master 127.13.162.62:43015 requested a full tablet report, sending...
I20260924 10:01:32.656574 14534 ts_manager.cc:194] Registered new tserver with Master: c29726605e0949d3a9cd4da3efa0d388 (127.13.162.3:37939)
I20260924 10:01:32.657334 14534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:46423
I20260924 10:01:32.664580 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:01:32.682168 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.670481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.13.162.62:43015)
0924 10:01:32.670635 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.670640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.671142 (+ 502us) server_negotiation.cc:407] Connection header received
0924 10:01:32.671257 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.671261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.671360 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.671450 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:32.672865 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.673605 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.674957 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.675379 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.675472 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.677643 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:32.677648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:32.677651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:32.677682 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:32.680034 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:32.680625 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:32.680629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:32.680631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:32.680750 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:32.681044 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:32.681047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:32.681049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:32.681336 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:32.681510 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.681762 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.681951 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:01:32.688344 14534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:32.695068 14534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:32.717736 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.707098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:37939 (local address 127.0.0.1:34268)
0924 10:01:32.707927 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.707977 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.708146 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.709376 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.709385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.709423 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:32.709822 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.709837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.712189 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.712195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.714692 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.714709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.716697 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.716704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.716956 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.717023 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.717134 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.717331 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":1663,"threads_started":1}
I20260924 10:01:32.719792 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.707380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.13.162.3:37939)
0924 10:01:32.708708 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.708717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.708751 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:01:32.708845 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.708852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.709137 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.710041 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:32.710302 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.712027 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.714926 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.716452 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.718909 (+ 2457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.719187 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.719265 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.719440 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":1014,"threads_started":1}
I20260924 10:01:32.724241 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 2244b7bf6f644933ad7f27fc71d4889f (DEFAULT_TABLE table=test-table [id=df49ad2c5d8f4e1ea7828c2f12406009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:32.725889 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2244b7bf6f644933ad7f27fc71d4889f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:32.730252 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.707232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41461 (local address 127.0.0.1:57622)
0924 10:01:32.709001 (+ 1769us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.709022 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.709132 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.722817 (+ 13685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.722822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.722847 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:32.723190 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.723202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.725150 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.725155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.727103 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.727118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.729408 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.729414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.729602 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.729636 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.729725 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.729901 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1664,"thread_start_us":667,"threads_started":1}
I20260924 10:01:32.730288 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.707743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.13.162.1:41461)
0924 10:01:32.721937 (+ 14194us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.721947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.721976 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:01:32.722074 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.722081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.722378 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.722563 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:32.723378 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.724993 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.727280 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.729115 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.729251 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.729459 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.729780 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.729984 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14031,"thread_start_us":13899,"threads_started":1}
I20260924 10:01:32.733726 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.708675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44255 (local address 127.0.0.1:41612)
0924 10:01:32.718380 (+ 9705us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.718401 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.718500 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.727827 (+ 9327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.727832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.727861 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:32.728060 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.728070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.730043 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.730048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.731681 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.731693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.733037 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.733043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.733205 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.733236 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.733314 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.733457 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9585,"thread_start_us":20469,"threads_started":1}
I20260924 10:01:32.733803 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.709144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.13.162.2:44255)
0924 10:01:32.727301 (+ 18157us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.727310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.727336 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:01:32.727423 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.727430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.727645 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.727852 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:32.728245 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.729877 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.731861 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.732880 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.733001 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.733185 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.733376 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.733521 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18000,"thread_start_us":18874,"threads_started":1}
I20260924 10:01:32.734287 14658 tablet_service.cc:1467] Processing CreateTablet for tablet 2244b7bf6f644933ad7f27fc71d4889f (DEFAULT_TABLE table=test-table [id=df49ad2c5d8f4e1ea7828c2f12406009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:32.735772 14658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2244b7bf6f644933ad7f27fc71d4889f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:32.736821 14792 tablet_service.cc:1467] Processing CreateTablet for tablet 2244b7bf6f644933ad7f27fc71d4889f (DEFAULT_TABLE table=test-table [id=df49ad2c5d8f4e1ea7828c2f12406009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:32.737964 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2244b7bf6f644933ad7f27fc71d4889f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:32.740510 15008 tablet_bootstrap.cc:492] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: Bootstrap starting.
I20260924 10:01:32.744079 15008 tablet_bootstrap.cc:654] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:32.745316 15008 log.cc:824] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:32.747855 15010 tablet_bootstrap.cc:492] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: Bootstrap starting.
I20260924 10:01:32.749619 15008 tablet_bootstrap.cc:492] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: No bootstrap required, opened a new log
I20260924 10:01:32.749907 15008 ts_tablet_manager.cc:1397] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 10:01:32.750901 15010 tablet_bootstrap.cc:654] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:32.751564 14723 heartbeater.cc:503] Master 127.13.162.62:43015 was elected leader, sending a full tablet report...
I20260924 10:01:32.751561 15011 tablet_bootstrap.cc:492] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: Bootstrap starting.
I20260924 10:01:32.752326 15010 log.cc:824] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:32.756316 15011 tablet_bootstrap.cc:654] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:32.757612 15011 log.cc:824] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:32.758365 15010 tablet_bootstrap.cc:492] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: No bootstrap required, opened a new log
I20260924 10:01:32.758685 15010 ts_tablet_manager.cc:1397] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260924 10:01:32.761165 15008 raft_consensus.cc:348] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.761651 15008 raft_consensus.cc:374] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:32.761791 15008 raft_consensus.cc:729] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29726605e0949d3a9cd4da3efa0d388, State: Initialized, Role: FOLLOWER
I20260924 10:01:32.762598 15008 consensus_queue.cc:260] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.763671 14990 heartbeater.cc:503] Master 127.13.162.62:43015 was elected leader, sending a full tablet report...
I20260924 10:01:32.764286 15008 ts_tablet_manager.cc:1428] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 10:01:32.764979 15011 tablet_bootstrap.cc:492] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: No bootstrap required, opened a new log
I20260924 10:01:32.765254 15011 ts_tablet_manager.cc:1397] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260924 10:01:32.767084 15010 raft_consensus.cc:348] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.767498 15010 raft_consensus.cc:374] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:32.767597 15010 raft_consensus.cc:729] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c12eab1bc04410aa38bd9f45085b0f5, State: Initialized, Role: FOLLOWER
I20260924 10:01:32.768154 15010 consensus_queue.cc:260] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.769716 15010 ts_tablet_manager.cc:1428] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260924 10:01:32.773557 15011 raft_consensus.cc:348] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.773890 15011 raft_consensus.cc:374] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:32.773968 15011 raft_consensus.cc:729] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac3cd624d3b48d9bd09d8ede4edcf91, State: Initialized, Role: FOLLOWER
I20260924 10:01:32.774459 15011 consensus_queue.cc:260] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.775332 14857 heartbeater.cc:503] Master 127.13.162.62:43015 was elected leader, sending a full tablet report...
I20260924 10:01:32.775708 15011 ts_tablet_manager.cc:1428] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260924 10:01:32.883627 14991 tablet.cc:2369] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:01:32.931057 14858 tablet.cc:2369] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:01:32.975440 14724 tablet.cc:2369] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:01:32.987216 15014 raft_consensus.cc:482] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:32.987525 15014 raft_consensus.cc:504] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:32.989230 15014 leader_election.cc:288] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c12eab1bc04410aa38bd9f45085b0f5 (127.13.162.1:41461), 3ac3cd624d3b48d9bd09d8ede4edcf91 (127.13.162.2:44255)
I20260924 10:01:32.995726 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.989449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41461 (local address 127.13.162.3:35665)
0924 10:01:32.989632 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.989650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.989775 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.990346 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.990351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.990378 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:32.990574 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.990583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.991826 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.991830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.993693 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.993708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.995120 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.995125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.995252 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.995277 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.995348 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.995447 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:01:32.995743 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.989668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:35665 (local address 127.13.162.1:41461)
0924 10:01:32.989909 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.989915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.989937 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:01:32.990020 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.990025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.990103 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.990203 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:32.990705 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.991696 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.993889 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.994985 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.995145 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.995227 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.995397 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.995522 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:01:32.996239 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.990080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46629 (local address 127.13.162.2:44255)
0924 10:01:32.990276 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:32.990281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:32.990539 (+ 258us) server_negotiation.cc:407] Connection header received
0924 10:01:32.990693 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:32.990698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:32.990769 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:32.990858 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:32.991258 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.992111 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.994240 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.995422 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.995562 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.995626 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:32.995954 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 10:01:32.996074 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:01:32.996821 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:32.989813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44255 (local address 127.13.162.3:46629)
0924 10:01:32.990450 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:32.990467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:32.990579 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:32.990866 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:32.990868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:32.990885 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:32.991130 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.991136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.992249 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.992254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.994101 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:32.994112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:32.995611 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:32.995617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:32.995778 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:32.995807 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:01:32.996395 (+ 588us) client_negotiation.cc:240] Negotiation successful
0924 10:01:32.996528 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":290,"threads_started":1}
I20260924 10:01:32.996855 14678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2244b7bf6f644933ad7f27fc71d4889f" candidate_uuid: "c29726605e0949d3a9cd4da3efa0d388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" is_pre_election: true
I20260924 10:01:32.997423 14678 raft_consensus.cc:2454] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c29726605e0949d3a9cd4da3efa0d388 in term 0.
I20260924 10:01:32.997903 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2244b7bf6f644933ad7f27fc71d4889f" candidate_uuid: "c29726605e0949d3a9cd4da3efa0d388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" is_pre_election: true
I20260924 10:01:32.998061 14881 leader_election.cc:302] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c12eab1bc04410aa38bd9f45085b0f5, c29726605e0949d3a9cd4da3efa0d388; no voters:
I20260924 10:01:32.998417 15014 raft_consensus.cc:2781] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:32.998407 14812 raft_consensus.cc:2454] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c29726605e0949d3a9cd4da3efa0d388 in term 0.
I20260924 10:01:32.998531 15014 raft_consensus.cc:482] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:32.998617 15014 raft_consensus.cc:3037] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:33.000834 15014 raft_consensus.cc:504] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:33.001466 15014 leader_election.cc:288] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [CANDIDATE]: Term 1 election: Requested vote from peers 6c12eab1bc04410aa38bd9f45085b0f5 (127.13.162.1:41461), 3ac3cd624d3b48d9bd09d8ede4edcf91 (127.13.162.2:44255)
I20260924 10:01:33.001837 14678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2244b7bf6f644933ad7f27fc71d4889f" candidate_uuid: "c29726605e0949d3a9cd4da3efa0d388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c12eab1bc04410aa38bd9f45085b0f5"
I20260924 10:01:33.001935 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2244b7bf6f644933ad7f27fc71d4889f" candidate_uuid: "c29726605e0949d3a9cd4da3efa0d388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91"
I20260924 10:01:33.002019 14678 raft_consensus.cc:3037] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:33.002143 14812 raft_consensus.cc:3037] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:33.004148 14678 raft_consensus.cc:2454] T 2244b7bf6f644933ad7f27fc71d4889f P 6c12eab1bc04410aa38bd9f45085b0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c29726605e0949d3a9cd4da3efa0d388 in term 1.
I20260924 10:01:33.004276 14812 raft_consensus.cc:2454] T 2244b7bf6f644933ad7f27fc71d4889f P 3ac3cd624d3b48d9bd09d8ede4edcf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c29726605e0949d3a9cd4da3efa0d388 in term 1.
I20260924 10:01:33.004508 14881 leader_election.cc:302] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c12eab1bc04410aa38bd9f45085b0f5, c29726605e0949d3a9cd4da3efa0d388; no voters:
I20260924 10:01:33.004769 15014 raft_consensus.cc:2781] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:33.005214 15014 raft_consensus.cc:686] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [term 1 LEADER]: Becoming Leader. State: Replica: c29726605e0949d3a9cd4da3efa0d388, State: Running, Role: LEADER
I20260924 10:01:33.005728 15014 consensus_queue.cc:237] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } }
I20260924 10:01:33.010012 14534 catalog_manager.cc:5495] T 2244b7bf6f644933ad7f27fc71d4889f P c29726605e0949d3a9cd4da3efa0d388 reported cstate change: term changed from 0 to 1, leader changed from <none> to c29726605e0949d3a9cd4da3efa0d388 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "c29726605e0949d3a9cd4da3efa0d388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29726605e0949d3a9cd4da3efa0d388" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 37939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c12eab1bc04410aa38bd9f45085b0f5" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ac3cd624d3b48d9bd09d8ede4edcf91" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44255 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:33.088737 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:33.075159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.13.162.1:41461)
0924 10:01:33.075334 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:33.075341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:33.075715 (+ 374us) server_negotiation.cc:407] Connection header received
0924 10:01:33.075873 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:33.075877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:33.075962 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:33.076046 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:33.077527 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.078277 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.079492 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.079907 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.080020 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:33.083745 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:33.083760 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:33.083776 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:33.083814 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:33.086209 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.086836 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.086843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.086846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.086976 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.087294 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.087298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.087300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.087652 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:33.087920 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:33.088206 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:01:33.088403 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244093, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244093, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:33.129591 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:33.116855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.13.162.1:41461)
0924 10:01:33.117039 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:33.117045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:33.117359 (+ 314us) server_negotiation.cc:407] Connection header received
0924 10:01:33.117530 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:33.117534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:33.117594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:33.117674 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:33.118973 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.119747 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.120989 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.121428 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.121537 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:33.125080 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:33.125086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:33.125089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:33.125122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:33.127343 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.127898 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.127902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.127904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.128024 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.128343 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.128367 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.128369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.128620 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:33.128813 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:33.129116 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 10:01:33.129279 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:33.167971 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:33.155498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.13.162.1:41461)
0924 10:01:33.155673 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:33.155680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:33.156005 (+ 325us) server_negotiation.cc:407] Connection header received
0924 10:01:33.156169 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:33.156173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:33.156236 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:33.156321 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:33.157597 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.158269 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.159467 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.159861 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.159977 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:33.163462 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:33.163468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:33.163471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:33.163502 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:33.165847 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.166408 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.166414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.166417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.166548 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.166870 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.166873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.166875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.167182 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:33.167368 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:33.167574 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 10:01:33.167732 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 10:01:33.169577 14638 tablet_service.cc:2908] Rejecting scan request for tablet 2244b7bf6f644933ad7f27fc71d4889f: Uninitialized: clean time has not yet been initialized
I20260924 10:01:33.172399 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14595
I20260924 10:01:33.185003 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:33.173492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.13.162.1:41461)
0924 10:01:33.173629 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:33.173633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:33.174003 (+ 370us) server_negotiation.cc:407] Connection header received
0924 10:01:33.174169 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:33.174173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:33.174246 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:33.174358 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:33.175425 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.176090 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.177391 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.177836 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.177959 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:33.180152 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:33.180157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:33.180160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:33.180190 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:33.182726 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.183316 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.183321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.183323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.183439 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.183752 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.183757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.183760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.184018 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:33.184280 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:33.184526 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 10:01:33.184728 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:01:33.185575 14718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:33.278177 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14595
I20260924 10:01:33.290951 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14729
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:01:33.304715 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:33.291675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.13.162.2:44255)
0924 10:01:33.291884 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:33.291891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:33.291921 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:01:33.292068 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:33.292072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:33.292142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:33.292228 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:33.293345 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.294098 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.295375 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.295755 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.295879 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:33.299610 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:33.299625 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:33.299640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:33.299674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:33.302191 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.302823 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.302831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.302833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.302958 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.303303 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.303307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.303309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.303660 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:33.303932 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:33.304208 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:01:33.304423 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:01:33.305275 14852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:33.390106 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14729
I20260924 10:01:33.402455 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14862
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:01:33.416018 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:33.403107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.13.162.3:37939)
0924 10:01:33.403594 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:33.403599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:33.403615 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:01:33.403664 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:33.403668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:33.403731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:33.403814 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:33.404971 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.405645 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.406900 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:33.407277 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:33.407366 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:33.411030 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:33.411043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:33.411058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:33.411094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:33.413452 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.414063 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.414070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.414072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.414196 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:33.414697 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:33.414700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:33.414702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:33.415023 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:33.415237 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:33.415534 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 10:01:33.415697 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":265,"threads_started":1}
I20260924 10:01:33.416647 14985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:33.503533 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14862
I20260924 10:01:33.516182 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 14502
I20260924 10:01:33.517364 14564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:33.632477 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 14502
2026-09-24T10:01:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3993 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[15039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[15039](info): set up 2 sockets
Sep 24 10:01:33 dist-test-slave-jvsj krb5kdc[15039](info): commencing operation
krb5kdc: starting...
W20260924 10:01:34.626166 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.007s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:34 dist-test-slave-jvsj krb5kdc[15039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:34.785981 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36027
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:35.088742 15052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:35.089039 15052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:35.089177 15052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:35.096328 15052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:35.096429 15052 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:01:35.096473 15052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:35.096556 15052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:35.096614 15052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:35.096693 15052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:35.106413 15052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:35.107647 15052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:35.108930 15052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:35.120175 15057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:35.120206 15060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:35.120230 15058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:35.121315 15052 server_base.cc:1034] running on GCE node
I20260924 10:01:35.122045 15052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:35.123431 15052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:35.124647 15052 hybrid_clock.cc:648] HybridClock initialized: now 1790244095124584 us; error 42 us; skew 500 ppm
I20260924 10:01:35.124955 15052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:35 dist-test-slave-jvsj krb5kdc[15039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244095, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:35.127997 15052 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:35.129374 15052 webserver.cc:460] Webserver started at http://127.13.162.62:41973/ using document root <none> and password file <none>
I20260924 10:01:35.130060 15052 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:35.130153 15052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:35.130489 15052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:35.133256 15052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "9902316c6a7740b9b346762e698b5517"
format_stamp: "Formatted at 2026-09-24 10:01:35 on dist-test-slave-jvsj"
server_key: "02a0e49e01ee75bdd0c1690f5083f47a"
server_key_iv: "11369681dc0d7062c5c30057b0fb8ee3"
server_key_version: "encryptionkey@0"
I20260924 10:01:35.134001 15052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "9902316c6a7740b9b346762e698b5517"
format_stamp: "Formatted at 2026-09-24 10:01:35 on dist-test-slave-jvsj"
server_key: "02a0e49e01ee75bdd0c1690f5083f47a"
server_key_iv: "11369681dc0d7062c5c30057b0fb8ee3"
server_key_version: "encryptionkey@0"
I20260924 10:01:35.138893 15052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 10:01:35.142097 15067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:35.143441 15052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:35.143589 15052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "9902316c6a7740b9b346762e698b5517"
format_stamp: "Formatted at 2026-09-24 10:01:35 on dist-test-slave-jvsj"
server_key: "02a0e49e01ee75bdd0c1690f5083f47a"
server_key_iv: "11369681dc0d7062c5c30057b0fb8ee3"
server_key_version: "encryptionkey@0"
I20260924 10:01:35.143817 15052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:35.155794 15052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:35.159677 15052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:35.159901 15052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:35.178864 15052 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:36027
I20260924 10:01:35.178900 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:36027 every 8 connection(s)
I20260924 10:01:35.180126 15052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:35.183683 15129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:35.190178 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15052
I20260924 10:01:35.190441 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:35.191223 13960 external_mini_cluster.cc:1428] Setting key 288aceb42bc45f97faeb43257aa9de50
I20260924 10:01:35.194404 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: Bootstrap starting.
I20260924 10:01:35.197422 15129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:35.198427 15129 log.cc:824] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:35.201068 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: No bootstrap required, opened a new log
Sep 24 10:01:35 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:35.208926 15129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9902316c6a7740b9b346762e698b5517" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36027 } }
I20260924 10:01:35.209285 15129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:35.209395 15129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9902316c6a7740b9b346762e698b5517, State: Initialized, Role: FOLLOWER
I20260924 10:01:35.210000 15129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9902316c6a7740b9b346762e698b5517" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36027 } }
I20260924 10:01:35.210284 15129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:35.210444 15129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:35.210618 15129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:35.213339 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:35.193217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.13.162.62:36027)
0924 10:01:35.194024 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:35.194040 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:35.194081 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:01:35.195824 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:35.195847 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:35.196240 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:35.196584 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:35.197953 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.199612 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.200787 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.201481 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.201644 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:35.205564 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:35.205572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:35.205591 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:35.205623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:35.208659 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:35.209300 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:35.209309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:35.209327 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:35.209545 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:35.210033 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:35.210038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:35.210041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:35.210809 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:35.211123 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:35.211500 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 10:01:35.211977 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":347,"threads_started":1}
I20260924 10:01:35.213745 15129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9902316c6a7740b9b346762e698b5517" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36027 } }
I20260924 10:01:35.214387 15129 leader_election.cc:302] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9902316c6a7740b9b346762e698b5517; no voters:
I20260924 10:01:35.215026 15129 leader_election.cc:288] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:35.215178 15134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:35.216233 15134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [term 1 LEADER]: Becoming Leader. State: Replica: 9902316c6a7740b9b346762e698b5517, State: Running, Role: LEADER
I20260924 10:01:35.216841 15134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9902316c6a7740b9b346762e698b5517" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36027 } }
I20260924 10:01:35.217162 15129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:35.219756 15135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9902316c6a7740b9b346762e698b5517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9902316c6a7740b9b346762e698b5517" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36027 } } }
I20260924 10:01:35.220116 15135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:35.222506 15136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9902316c6a7740b9b346762e698b5517. Latest consensus state: current_term: 1 leader_uuid: "9902316c6a7740b9b346762e698b5517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9902316c6a7740b9b346762e698b5517" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36027 } } }
I20260924 10:01:35.222724 15136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:35.223238 15142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:35.227524 15142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:35.235510 15142 catalog_manager.cc:1347] Generated new cluster ID: 3aaf99fa28324673b90781568aeac642
I20260924 10:01:35.235637 15142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:35.264967 15142 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:35.266840 15142 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:35.274677 15142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: Generated new TSK 0
I20260924 10:01:35.275708 15142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:35.323738 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36027
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:35.631111 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:35.631386 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:35.631489 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:35.638633 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:35.638736 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:35.638823 15155 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:01:35.638900 15155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:35.648635 15155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:36027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:35.649861 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:35.651182 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:35.663378 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:35.663551 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:35.663861 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:35.664044 15155 server_base.cc:1034] running on GCE node
I20260924 10:01:35.664655 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:35.665496 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:35.666702 15155 hybrid_clock.cc:648] HybridClock initialized: now 1790244095666637 us; error 47 us; skew 500 ppm
I20260924 10:01:35.666975 15155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:35 dist-test-slave-jvsj krb5kdc[15039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244095, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:35.670079 15155 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:35.671633 15155 webserver.cc:460] Webserver started at http://127.13.162.1:39119/ using document root <none> and password file <none>
I20260924 10:01:35.672288 15155 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:35.672384 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:35.672685 15155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:35.675475 15155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "6a9ae7c7eef8401ebd0f19b45106f29d"
format_stamp: "Formatted at 2026-09-24 10:01:35 on dist-test-slave-jvsj"
server_key: "534b39a0393e09d1d13a9a4a185d44fc"
server_key_iv: "f5447f48367526943c6617d5675db353"
server_key_version: "encryptionkey@0"
I20260924 10:01:35.676213 15155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "6a9ae7c7eef8401ebd0f19b45106f29d"
format_stamp: "Formatted at 2026-09-24 10:01:35 on dist-test-slave-jvsj"
server_key: "534b39a0393e09d1d13a9a4a185d44fc"
server_key_iv: "f5447f48367526943c6617d5675db353"
server_key_version: "encryptionkey@0"
I20260924 10:01:35.681152 15155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:01:35.684418 15170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:35.685675 15155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:35.685808 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "6a9ae7c7eef8401ebd0f19b45106f29d"
format_stamp: "Formatted at 2026-09-24 10:01:35 on dist-test-slave-jvsj"
server_key: "534b39a0393e09d1d13a9a4a185d44fc"
server_key_iv: "f5447f48367526943c6617d5675db353"
server_key_version: "encryptionkey@0"
I20260924 10:01:35.686002 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:35.712409 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:35.716610 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:35.716873 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:35.719549 15155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:35.719647 15155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:35.719780 15155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:35.719846 15155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:01:35 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244095, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:35.741892 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:35.720166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52211 (local address 127.13.162.62:36027)
0924 10:01:35.720611 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:35.720618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:35.722311 (+ 1693us) server_negotiation.cc:407] Connection header received
0924 10:01:35.724228 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:35.724236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:35.724319 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:35.724447 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:35.726685 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.727649 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.730143 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.730638 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.730780 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:35.735673 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:35.735679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:35.735682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:35.735714 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:35.738285 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:35.739350 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:35.739356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:35.739360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:35.739492 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:35.740057 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:35.740062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:35.740065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:35.740426 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:35.740555 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:35.741334 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 10:01:35.741546 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":221,"threads_started":1}
I20260924 10:01:35.743209 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:35.720978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.13.162.1:52211)
0924 10:01:35.721868 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:35.721929 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:35.723933 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:35.724732 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:35.724740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:35.725577 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:35.726496 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:35.726526 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.727833 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.727839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:35.729996 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:35.730009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.730892 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.730901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:35.731237 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:35.732052 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:35.732088 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:35.735429 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:35.738458 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:35.738471 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:35.738494 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:35.739197 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:35.739607 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:35.739613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:35.739617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:35.739915 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:35.740552 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:35.740559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:35.740959 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 10:01:35.741202 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 10:01:35.741728 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":315,"threads_started":1}
I20260924 10:01:35.760303 15155 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:42043
I20260924 10:01:35.760339 15290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:42043 every 8 connection(s)
I20260924 10:01:35.761597 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:35.763567 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:35.749909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51389 (local address 127.13.162.62:36027)
0924 10:01:35.750092 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:35.750098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:35.750682 (+ 584us) server_negotiation.cc:407] Connection header received
0924 10:01:35.750858 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:35.750862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:35.750923 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:35.751008 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:35.752212 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.753248 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.755211 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.755808 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.755932 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:35.758389 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:35.758396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:35.758401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:35.758434 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:35.760987 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:35.761603 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:35.761609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:35.761613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:35.761762 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:35.762099 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:35.762104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:35.762108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:35.762465 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:35.762599 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:35.763024 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 10:01:35.763237 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:01:35.763621 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:35.749930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.13.162.1:51389)
0924 10:01:35.750573 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:35.750594 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:35.750749 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:35.751060 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:35.751065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:35.751600 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:35.752046 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:35.752060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.753434 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.753439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:35.755062 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:35.755080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:35.756166 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:35.756170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:35.756331 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:35.756997 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:35.757021 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:35.758133 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:35.761103 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:35.761108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:35.761113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:35.761501 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:35.761871 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:35.761876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:35.761880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:35.762006 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:35.762570 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:35.762574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:35.762782 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 10:01:35.762930 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 10:01:35.763210 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260924 10:01:35.763736 15291 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36027
I20260924 10:01:35.763936 15291 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:35.764645 15291 heartbeater.cc:511] Master 127.13.162.62:36027 requested a full tablet report, sending...
I20260924 10:01:35.766548 15084 ts_manager.cc:194] Registered new tserver with Master: 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043)
I20260924 10:01:35.767905 15084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:52211
I20260924 10:01:35.770140 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15155
I20260924 10:01:35.770366 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:35.770954 13960 external_mini_cluster.cc:1428] Setting key 7961138a131423fbfb10b06032776ed6
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:35.805496 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36027
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:36.110188 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:36.110464 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:36.110566 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:36.117578 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:36.117700 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:36.117818 15295 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:01:36.117936 15295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:36.127759 15295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:36027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:36.129061 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:36.130460 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:36.142601 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:36.142601 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:36.143186 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:36.143573 15295 server_base.cc:1034] running on GCE node
I20260924 10:01:36.144153 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:36.145056 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:36.146279 15295 hybrid_clock.cc:648] HybridClock initialized: now 1790244096146199 us; error 34 us; skew 500 ppm
I20260924 10:01:36.146631 15295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:36 dist-test-slave-jvsj krb5kdc[15039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:36.149663 15295 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:36.151100 15295 webserver.cc:460] Webserver started at http://127.13.162.2:35633/ using document root <none> and password file <none>
I20260924 10:01:36.151760 15295 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:36.151856 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:36.152153 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:36.154966 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "817571f429ab4aafb7d2cebfc141187c"
format_stamp: "Formatted at 2026-09-24 10:01:36 on dist-test-slave-jvsj"
server_key: "373e618311fb2d822794d16e28f3cc70"
server_key_iv: "8d7ada1dcb05ef16d5127a548374b0c5"
server_key_version: "encryptionkey@0"
I20260924 10:01:36.155697 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "817571f429ab4aafb7d2cebfc141187c"
format_stamp: "Formatted at 2026-09-24 10:01:36 on dist-test-slave-jvsj"
server_key: "373e618311fb2d822794d16e28f3cc70"
server_key_iv: "8d7ada1dcb05ef16d5127a548374b0c5"
server_key_version: "encryptionkey@0"
I20260924 10:01:36.160840 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:01:36.164266 15310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:36.165570 15295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:36.165711 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "817571f429ab4aafb7d2cebfc141187c"
format_stamp: "Formatted at 2026-09-24 10:01:36 on dist-test-slave-jvsj"
server_key: "373e618311fb2d822794d16e28f3cc70"
server_key_iv: "8d7ada1dcb05ef16d5127a548374b0c5"
server_key_version: "encryptionkey@0"
I20260924 10:01:36.165890 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:36.206743 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:36.211083 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:36.211329 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:36.213832 15295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:36.213923 15295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:36.214057 15295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:36.214115 15295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:01:36 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:36.236826 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.214545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:43895 (local address 127.13.162.62:36027)
0924 10:01:36.214717 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.214723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.216084 (+ 1361us) server_negotiation.cc:407] Connection header received
0924 10:01:36.218701 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.218706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.218775 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.218832 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:36.221059 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.222168 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.224359 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.224924 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.225051 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.230015 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:36.230022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:36.230027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:36.230066 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:36.233098 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.234022 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.234028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.234032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.234161 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.234740 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.234746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.234749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.235067 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:36.235213 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.236237 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.236466 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:01:36.238590 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.215099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.13.162.2:43895)
0924 10:01:36.215910 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.215957 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.218296 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.219330 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.219341 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.220085 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:36.220866 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.220888 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.222364 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.222370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.224232 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.224245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.225162 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.225170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.225505 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.226302 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:36.226358 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:36.229772 (+ 3414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:36.233264 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.233278 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.233299 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.233910 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.234271 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.234277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.234282 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.234602 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.235204 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:36.235217 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:36.235761 (+ 544us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.236118 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.236778 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":338,"threads_started":1}
I20260924 10:01:36.255280 15295 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:41811
I20260924 10:01:36.255300 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:41811 every 8 connection(s)
I20260924 10:01:36.256820 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:36.258852 15430 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36027
I20260924 10:01:36.259089 15430 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:36.259824 15430 heartbeater.cc:511] Master 127.13.162.62:36027 requested a full tablet report, sending...
I20260924 10:01:36.261212 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.244331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.13.162.2:37943)
0924 10:01:36.247007 (+ 2676us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.247027 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.247211 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.247699 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.247703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.248199 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:36.248563 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.248576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.250128 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.250134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.252339 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.252362 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.253346 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.253351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.253469 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.254107 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:36.254129 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:36.255060 (+ 931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:36.258523 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.258528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.258532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.259024 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.259429 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.259433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.259437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.259583 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.260231 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:36.260235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:36.260438 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.260571 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.260841 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2540,"thread_start_us":2427,"threads_started":1}
I20260924 10:01:36.261212 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.244245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:37943 (local address 127.13.162.62:36027)
0924 10:01:36.244406 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.244412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.247105 (+ 2693us) server_negotiation.cc:407] Connection header received
0924 10:01:36.247360 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.247366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.247453 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.247519 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:36.248711 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.249946 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.252537 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.253165 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.253284 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.255259 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:36.255267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:36.255271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:36.255306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:36.258403 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.259141 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.259147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.259150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.259288 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.259745 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.259749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.259751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.260120 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:36.260240 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.260715 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.260913 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:01:36.261336 15084 ts_manager.cc:194] Registered new tserver with Master: 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811)
I20260924 10:01:36.262461 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15295
I20260924 10:01:36.262521 15084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:43895
I20260924 10:01:36.262665 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:36.263216 13960 external_mini_cluster.cc:1428] Setting key 1d144ba93bd107a80dbefb4402d9e65a
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:36.297928 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36027
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:36.603483 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:36.603788 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:36.603935 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:36.611155 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:36.611275 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:36.611400 15434 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:01:36.611531 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:36.621209 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:36027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:36.622524 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:36.623785 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:36.635826 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:36.635897 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:36.635841 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:36.641032 15434 server_base.cc:1034] running on GCE node
I20260924 10:01:36.641901 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:36.643189 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:36.644394 15434 hybrid_clock.cc:648] HybridClock initialized: now 1790244096644315 us; error 55 us; skew 500 ppm
I20260924 10:01:36.644783 15434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:36 dist-test-slave-jvsj krb5kdc[15039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:36.648762 15434 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:36.650173 15434 webserver.cc:460] Webserver started at http://127.13.162.3:46101/ using document root <none> and password file <none>
I20260924 10:01:36.650911 15434 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:36.651011 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:36.651319 15434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:36.654168 15434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "421c91a0052d402da4d53553650fff6c"
format_stamp: "Formatted at 2026-09-24 10:01:36 on dist-test-slave-jvsj"
server_key: "738df3db54eeb72f059f0718c957ef67"
server_key_iv: "1bc637485c335b8e17a6f30937a3b26b"
server_key_version: "encryptionkey@0"
I20260924 10:01:36.655036 15434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "421c91a0052d402da4d53553650fff6c"
format_stamp: "Formatted at 2026-09-24 10:01:36 on dist-test-slave-jvsj"
server_key: "738df3db54eeb72f059f0718c957ef67"
server_key_iv: "1bc637485c335b8e17a6f30937a3b26b"
server_key_version: "encryptionkey@0"
I20260924 10:01:36.660003 15434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:01:36.663447 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:36.664736 15434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:01:36.664863 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "421c91a0052d402da4d53553650fff6c"
format_stamp: "Formatted at 2026-09-24 10:01:36 on dist-test-slave-jvsj"
server_key: "738df3db54eeb72f059f0718c957ef67"
server_key_iv: "1bc637485c335b8e17a6f30937a3b26b"
server_key_version: "encryptionkey@0"
I20260924 10:01:36.665030 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:36.679205 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:36.682859 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:36.683063 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:36.685459 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:36.685534 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:36.685655 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:36.685705 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:01:36 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:36.706983 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.686014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:33375 (local address 127.13.162.62:36027)
0924 10:01:36.686186 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.686192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.687686 (+ 1494us) server_negotiation.cc:407] Connection header received
0924 10:01:36.689760 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.689767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.689854 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.689916 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:36.692207 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.693163 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.695444 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.695967 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.696088 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.700518 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:36.700526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:36.700531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:36.700575 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:36.703586 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.704258 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.704264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.704268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.704433 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.704941 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.704946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.704950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.705298 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:36.705439 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.706429 (+ 990us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.706647 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:01:36.708000 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.686680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.13.162.3:33375)
0924 10:01:36.687502 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.687548 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.689407 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.690452 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.690463 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.691176 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:36.692025 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.692051 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.693355 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.693359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.695283 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.695315 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.696219 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.696226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.696429 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.697234 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:36.697270 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:36.700297 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:36.703721 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.703730 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.703747 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.704143 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.704561 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.704566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.704570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.704824 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.705427 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:36.705435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:36.705898 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.706238 (+ 340us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.706819 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":331,"threads_started":1}
I20260924 10:01:36.726862 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.714138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:58313 (local address 127.13.162.62:36027)
0924 10:01:36.714294 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.714299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.714746 (+ 447us) server_negotiation.cc:407] Connection header received
0924 10:01:36.714894 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.714900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.714965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.715039 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:36.715872 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.716553 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.717908 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.718492 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.718606 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.720856 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:36.720862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:36.720867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:36.720902 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:36.724194 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.725024 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.725028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.725030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.725146 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.725507 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.725510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.725512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.725793 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:36.725925 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.726394 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.726619 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:01:36.727169 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.714105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.13.162.3:58313)
0924 10:01:36.714658 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.714672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.714791 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.715072 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.715076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.715479 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:36.715746 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.715756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.716665 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.716670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.717758 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.717769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.718814 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.718820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.719027 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.719638 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:36.719661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:36.720687 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:36.724329 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.724336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.724341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.724908 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.725245 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:36.725251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:36.725255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:36.725407 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:36.725895 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:36.725901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:36.726118 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.726274 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.726670 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":315,"threads_started":1}
I20260924 10:01:36.735908 15434 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:41881
I20260924 10:01:36.735929 15568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:41881 every 8 connection(s)
I20260924 10:01:36.737053 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:36.738874 15569 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36027
I20260924 10:01:36.739082 15569 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:36.739687 15569 heartbeater.cc:511] Master 127.13.162.62:36027 requested a full tablet report, sending...
I20260924 10:01:36.740988 15084 ts_manager.cc:194] Registered new tserver with Master: 421c91a0052d402da4d53553650fff6c (127.13.162.3:41881)
I20260924 10:01:36.741819 15084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:33375
I20260924 10:01:36.744071 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15434
I20260924 10:01:36.744216 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:36.744573 13960 external_mini_cluster.cc:1428] Setting key 59a7d9f17ec49d052fb52d32e37dc54d
I20260924 10:01:36.747555 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:01:36 dist-test-slave-jvsj krb5kdc[15039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260924 10:01:36.770562 15291 heartbeater.cc:503] Master 127.13.162.62:36027 was elected leader, sending a full tablet report...
Sep 24 10:01:36 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:36.787869 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.775596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.13.162.62:36027)
0924 10:01:36.775790 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.775796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.775957 (+ 161us) server_negotiation.cc:407] Connection header received
0924 10:01:36.776105 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.776108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.776164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.776232 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:36.777538 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.778257 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.779461 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.779860 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.779950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.783425 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:36.783431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:36.783434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:36.783464 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:36.785688 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.786242 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.786246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.786248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.786420 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:36.786699 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:36.786702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:36.786704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:36.786962 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:36.787135 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.787420 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.787581 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:01:36.791785 15084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:36.798414 15084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:36.828464 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.809589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.13.162.1:42043)
0924 10:01:36.821251 (+ 11662us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.821262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.821294 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:01:36.821409 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.821417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.821714 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.822681 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:36.822875 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.824102 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.826422 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.827346 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.827470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.827687 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.827998 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.828165 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11472,"thread_start_us":11343,"threads_started":1}
I20260924 10:01:36.828511 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.809356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:42043 (local address 127.0.0.1:47930)
0924 10:01:36.810295 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.810372 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.810536 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.822000 (+ 11464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.822008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.822042 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:36.822466 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.822480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.824225 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.824230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.826267 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.826278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.827538 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.827544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.827745 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.827793 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.827923 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.828113 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":34,"thread_start_us":477,"threads_started":1}
I20260924 10:01:36.830435 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.810789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:41811 (local address 127.0.0.1:57740)
0924 10:01:36.821570 (+ 10781us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.821586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.821698 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.822952 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.822956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.822979 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:36.823313 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.823324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.824936 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.824940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.828003 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.828015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.829674 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.829680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.829838 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.829865 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.829941 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.830079 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10671,"thread_start_us":10521,"threads_started":1}
I20260924 10:01:36.830435 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.811305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.13.162.2:41811)
0924 10:01:36.822134 (+ 10829us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.822143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.822170 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:01:36.822264 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.822270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.822567 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.822755 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:36.823502 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.824794 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.828157 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.829493 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.829611 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.829796 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.829991 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.830147 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10670,"thread_start_us":11064,"threads_started":1}
I20260924 10:01:36.831372 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.809873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41881 (local address 127.0.0.1:39254)
0924 10:01:36.820399 (+ 10526us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.820421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.820569 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.823527 (+ 2958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.823531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.823554 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:36.823851 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.823859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.825663 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.825668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.827673 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.827687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.830818 (+ 3131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.830825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.830949 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.830971 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.831021 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.831102 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10373,"thread_start_us":10773,"threads_started":1}
I20260924 10:01:36.831935 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.810547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.13.162.3:41881)
0924 10:01:36.823040 (+ 12493us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.823048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.823080 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:01:36.823162 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.823169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.823378 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.823541 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:36.824020 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.825542 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.827807 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.829958 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.831230 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.831437 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.831501 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.831638 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12320,"thread_start_us":12155,"threads_started":1}
I20260924 10:01:36.833972 15360 tablet_service.cc:1467] Processing CreateTablet for tablet 27365fb9625f408b946004be68ce0765 (DEFAULT_TABLE table=test-table [id=5c0840b6c57c4e40aa282f06591e05d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:36.832780 15221 tablet_service.cc:1467] Processing CreateTablet for tablet 27365fb9625f408b946004be68ce0765 (DEFAULT_TABLE table=test-table [id=5c0840b6c57c4e40aa282f06591e05d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:36.834619 15499 tablet_service.cc:1467] Processing CreateTablet for tablet 27365fb9625f408b946004be68ce0765 (DEFAULT_TABLE table=test-table [id=5c0840b6c57c4e40aa282f06591e05d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:36.835139 15360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27365fb9625f408b946004be68ce0765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:36.835863 15221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27365fb9625f408b946004be68ce0765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:36.835876 15499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27365fb9625f408b946004be68ce0765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:36.845922 15588 tablet_bootstrap.cc:492] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Bootstrap starting.
I20260924 10:01:36.849038 15588 tablet_bootstrap.cc:654] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:36.849583 15590 tablet_bootstrap.cc:492] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Bootstrap starting.
I20260924 10:01:36.850805 15588 log.cc:824] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:36.851400 15589 tablet_bootstrap.cc:492] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Bootstrap starting.
I20260924 10:01:36.853134 15590 tablet_bootstrap.cc:654] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:36.853888 15588 tablet_bootstrap.cc:492] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: No bootstrap required, opened a new log
I20260924 10:01:36.854079 15590 log.cc:824] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:36.854146 15588 ts_tablet_manager.cc:1397] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 10:01:36.854398 15589 tablet_bootstrap.cc:654] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:36.855531 15589 log.cc:824] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:36.856582 15590 tablet_bootstrap.cc:492] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: No bootstrap required, opened a new log
I20260924 10:01:36.856848 15590 ts_tablet_manager.cc:1397] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:01:36.858276 15589 tablet_bootstrap.cc:492] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: No bootstrap required, opened a new log
I20260924 10:01:36.859823 15589 ts_tablet_manager.cc:1397] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260924 10:01:36.863654 15590 raft_consensus.cc:348] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.864007 15590 raft_consensus.cc:374] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:36.863808 15588 raft_consensus.cc:348] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.864142 15590 raft_consensus.cc:729] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9ae7c7eef8401ebd0f19b45106f29d, State: Initialized, Role: FOLLOWER
I20260924 10:01:36.864219 15588 raft_consensus.cc:374] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:36.864312 15588 raft_consensus.cc:729] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817571f429ab4aafb7d2cebfc141187c, State: Initialized, Role: FOLLOWER
I20260924 10:01:36.864847 15590 consensus_queue.cc:260] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.864886 15588 consensus_queue.cc:260] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.866565 15590 ts_tablet_manager.cc:1428] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 10:01:36.867559 15430 heartbeater.cc:503] Master 127.13.162.62:36027 was elected leader, sending a full tablet report...
I20260924 10:01:36.868088 15588 ts_tablet_manager.cc:1428] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260924 10:01:36.870114 15589 raft_consensus.cc:348] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.870615 15589 raft_consensus.cc:374] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:36.870750 15589 raft_consensus.cc:729] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421c91a0052d402da4d53553650fff6c, State: Initialized, Role: FOLLOWER
I20260924 10:01:36.871439 15589 consensus_queue.cc:260] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.872511 15569 heartbeater.cc:503] Master 127.13.162.62:36027 was elected leader, sending a full tablet report...
I20260924 10:01:36.873013 15589 ts_tablet_manager.cc:1428] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 10:01:36.899619 15596 raft_consensus.cc:482] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:36.899881 15596 raft_consensus.cc:504] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.901397 15596 leader_election.cc:288] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043), 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811)
I20260924 10:01:36.906548 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.901813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:56717 (local address 127.13.162.1:42043)
0924 10:01:36.902033 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.902039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.902059 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:01:36.902207 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.902217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.902288 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.902394 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:36.902783 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.903472 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.904997 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.905727 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.905865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.905936 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.906171 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.906289 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:01:36.906548 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.901696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:42043 (local address 127.13.162.3:56717)
0924 10:01:36.901937 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.901955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.902047 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.902433 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.902437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.902469 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:36.902660 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.902668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.903592 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.903596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.904856 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.904867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.905865 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.905871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.906039 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.906061 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.906137 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.906261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":21}
I20260924 10:01:36.907501 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.902021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:41811 (local address 127.13.162.3:36401)
0924 10:01:36.902708 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:36.902721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:36.902829 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:36.903113 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:36.903117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:36.903140 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:36.903435 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.903443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.904369 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.904374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.905686 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:36.905698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.906879 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.906885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:36.907043 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.907074 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:01:36.907159 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:01:36.907246 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":1,"thread_start_us":470,"threads_started":1}
I20260924 10:01:36.907599 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.902288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:36401 (local address 127.13.162.2:41811)
0924 10:01:36.902483 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.902488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.902799 (+ 311us) server_negotiation.cc:407] Connection header received
0924 10:01:36.902967 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.902970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.903023 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.903092 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:36.903568 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.904248 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.905828 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.906595 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.906746 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.906805 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.907216 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.907340 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:01:36.907619 15241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27365fb9625f408b946004be68ce0765" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" is_pre_election: true
I20260924 10:01:36.908133 15241 raft_consensus.cc:2454] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 0.
I20260924 10:01:36.908545 15380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27365fb9625f408b946004be68ce0765" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817571f429ab4aafb7d2cebfc141187c" is_pre_election: true
I20260924 10:01:36.908789 15450 leader_election.cc:302] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 421c91a0052d402da4d53553650fff6c, 6a9ae7c7eef8401ebd0f19b45106f29d; no voters:
I20260924 10:01:36.908955 15380 raft_consensus.cc:2454] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 0.
I20260924 10:01:36.909176 15596 raft_consensus.cc:2781] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:36.909319 15596 raft_consensus.cc:482] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:36.909449 15596 raft_consensus.cc:3037] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:36.911752 15596 raft_consensus.cc:504] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.912243 15596 leader_election.cc:288] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 election: Requested vote from peers 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043), 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811)
I20260924 10:01:36.912727 15380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27365fb9625f408b946004be68ce0765" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817571f429ab4aafb7d2cebfc141187c"
I20260924 10:01:36.912719 15241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27365fb9625f408b946004be68ce0765" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d"
I20260924 10:01:36.912915 15380 raft_consensus.cc:3037] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:36.912926 15241 raft_consensus.cc:3037] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:36.915951 15241 raft_consensus.cc:2454] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 1.
I20260924 10:01:36.915951 15380 raft_consensus.cc:2454] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 1.
I20260924 10:01:36.916294 15450 leader_election.cc:302] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 421c91a0052d402da4d53553650fff6c, 6a9ae7c7eef8401ebd0f19b45106f29d; no voters:
I20260924 10:01:36.916545 15596 raft_consensus.cc:2781] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:36.916957 15596 raft_consensus.cc:686] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 1 LEADER]: Becoming Leader. State: Replica: 421c91a0052d402da4d53553650fff6c, State: Running, Role: LEADER
I20260924 10:01:36.917462 15596 consensus_queue.cc:237] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:36.922088 15083 catalog_manager.cc:5495] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 421c91a0052d402da4d53553650fff6c (127.13.162.3). New cstate: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } health_report { overall_health: HEALTHY } } }
I20260924 10:01:36.959986 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:36.954987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.13.162.3:41881)
0924 10:01:36.955173 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:36.955178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:36.955196 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:01:36.955482 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:36.955486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:36.955571 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:36.955686 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:36.956024 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.956689 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.957964 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:36.958406 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:36.958509 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:36.958956 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:36.959425 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:36.959546 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:36.959670 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:01:36.959745 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:01:36.967653 15380 raft_consensus.cc:1261] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Refusing update from remote peer 421c91a0052d402da4d53553650fff6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:36.967648 15241 raft_consensus.cc:1261] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Refusing update from remote peer 421c91a0052d402da4d53553650fff6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:36.968562 15598 consensus_queue.cc:1035] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:36.968845 15596 consensus_queue.cc:1035] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:36.980242 15601 mvcc.cc:205] Tried to move back new op lower bound from 7332839821170061312 to 7332839820978118656. Current Snapshot: MvccSnapshot[applied={T|T < 7332839821170061312}]
I20260924 10:01:36.981153 15602 mvcc.cc:205] Tried to move back new op lower bound from 7332839821170061312 to 7332839820978118656. Current Snapshot: MvccSnapshot[applied={T|T < 7332839821170061312}]
I20260924 10:01:36.998502 15084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:36.998787 15084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:37.002295 15084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: Sending DeleteTablet for 3 replicas of tablet 27365fb9625f408b946004be68ce0765
I20260924 10:01:37.003489 15499 tablet_service.cc:1514] Processing DeleteTablet for tablet 27365fb9625f408b946004be68ce0765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:36 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:39254
I20260924 10:01:37.003511 15360 tablet_service.cc:1514] Processing DeleteTablet for tablet 27365fb9625f408b946004be68ce0765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:36 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57740
I20260924 10:01:37.003613 15221 tablet_service.cc:1514] Processing DeleteTablet for tablet 27365fb9625f408b946004be68ce0765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:36 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:47930
I20260924 10:01:37.005875 15084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:37.007367 15084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:37.016798 15498 tablet_service.cc:1467] Processing CreateTablet for tablet 470e8e9ea34e428facbf7e477a9ec13d (DEFAULT_TABLE table=test-table [id=3a542828334349cf982030e6fca6eca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:37.017231 15220 tablet_service.cc:1467] Processing CreateTablet for tablet 470e8e9ea34e428facbf7e477a9ec13d (DEFAULT_TABLE table=test-table [id=3a542828334349cf982030e6fca6eca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:37.017405 15498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470e8e9ea34e428facbf7e477a9ec13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:37.017786 15220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470e8e9ea34e428facbf7e477a9ec13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:37.023073 15359 tablet_service.cc:1467] Processing CreateTablet for tablet 470e8e9ea34e428facbf7e477a9ec13d (DEFAULT_TABLE table=test-table [id=3a542828334349cf982030e6fca6eca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:37.023649 15359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470e8e9ea34e428facbf7e477a9ec13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:37.024501 15590 tablet_bootstrap.cc:492] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Bootstrap starting.
I20260924 10:01:37.026193 15590 tablet_bootstrap.cc:654] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:37.030845 15590 tablet_bootstrap.cc:492] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: No bootstrap required, opened a new log
I20260924 10:01:37.031008 15590 ts_tablet_manager.cc:1397] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 10:01:37.031813 15590 raft_consensus.cc:348] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.032119 15590 raft_consensus.cc:374] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:37.032261 15590 raft_consensus.cc:729] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9ae7c7eef8401ebd0f19b45106f29d, State: Initialized, Role: FOLLOWER
I20260924 10:01:37.032598 15590 consensus_queue.cc:260] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.035343 15590 ts_tablet_manager.cc:1428] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:01:37.038731 15588 tablet_bootstrap.cc:492] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Bootstrap starting.
I20260924 10:01:37.039250 15589 tablet_bootstrap.cc:492] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Bootstrap starting.
I20260924 10:01:37.040372 15588 tablet_bootstrap.cc:654] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:37.040876 15589 tablet_bootstrap.cc:654] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:37.049968 15594 raft_consensus.cc:482] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:37.050200 15594 raft_consensus.cc:504] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.052096 15594 leader_election.cc:288] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811), 421c91a0052d402da4d53553650fff6c (127.13.162.3:41881)
I20260924 10:01:37.052498 15588 tablet_bootstrap.cc:492] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: No bootstrap required, opened a new log
I20260924 10:01:37.052699 15588 ts_tablet_manager.cc:1397] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20260924 10:01:37.053609 15588 raft_consensus.cc:348] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.054015 15588 raft_consensus.cc:374] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:37.054154 15588 raft_consensus.cc:729] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817571f429ab4aafb7d2cebfc141187c, State: Initialized, Role: FOLLOWER
I20260924 10:01:37.054487 15588 consensus_queue.cc:260] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.057305 15588 ts_tablet_manager.cc:1428] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 10:01:37.070595 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.053775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:57231 (local address 127.13.162.2:41811)
0924 10:01:37.053913 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.053919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.060016 (+ 6097us) server_negotiation.cc:407] Connection header received
0924 10:01:37.060281 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.060287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.060355 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.060429 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:37.061543 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.062719 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.067586 (+ 4867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.068776 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.068910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.068981 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.070115 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.070252 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:01:37.071501 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.052644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:41811 (local address 127.13.162.1:57231)
0924 10:01:37.059922 (+ 7278us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:37.059941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:37.060129 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:37.060628 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:37.060633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:37.060658 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:37.060979 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.060992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.062908 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.062914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.065070 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.065085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.069248 (+ 4163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.069254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.069707 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.069740 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:01:37.071050 (+ 1310us) client_negotiation.cc:240] Negotiation successful
0924 10:01:37.071190 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7160,"threads_started":1}
I20260924 10:01:37.082145 15380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470e8e9ea34e428facbf7e477a9ec13d" candidate_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817571f429ab4aafb7d2cebfc141187c" is_pre_election: true
I20260924 10:01:37.082434 15380 raft_consensus.cc:2454] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a9ae7c7eef8401ebd0f19b45106f29d in term 0.
I20260924 10:01:37.082497 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.053496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:38271 (local address 127.13.162.3:41881)
0924 10:01:37.053659 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.053665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.072502 (+ 18837us) server_negotiation.cc:407] Connection header received
0924 10:01:37.073111 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.073117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.073192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.073248 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:37.074629 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.075740 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.079617 (+ 3877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.080878 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.081010 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.081086 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.082016 (+ 930us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.082155 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:01:37.082890 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.053123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41881 (local address 127.13.162.1:38271)
0924 10:01:37.072413 (+ 19290us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:37.072433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:37.072851 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:37.073449 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:37.073454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:37.073478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:37.074305 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.074333 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.075933 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.075938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.077599 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.077612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.081259 (+ 3647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.081265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.081440 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.081474 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:01:37.082485 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0924 10:01:37.082617 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19162,"thread_start_us":6525,"threads_started":1}
I20260924 10:01:37.083349 15589 tablet_bootstrap.cc:492] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: No bootstrap required, opened a new log
I20260924 10:01:37.083264 15171 leader_election.cc:302] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9ae7c7eef8401ebd0f19b45106f29d, 817571f429ab4aafb7d2cebfc141187c; no voters:
I20260924 10:01:37.084163 15594 raft_consensus.cc:2781] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:37.084290 15594 raft_consensus.cc:482] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:37.084379 15594 raft_consensus.cc:3037] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:37.087668 15594 raft_consensus.cc:504] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.088960 15594 leader_election.cc:288] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [CANDIDATE]: Term 1 election: Requested vote from peers 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811), 421c91a0052d402da4d53553650fff6c (127.13.162.3:41881)
I20260924 10:01:37.083572 15589 ts_tablet_manager.cc:1397] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.007s
I20260924 10:01:37.091179 15380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470e8e9ea34e428facbf7e477a9ec13d" candidate_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817571f429ab4aafb7d2cebfc141187c"
I20260924 10:01:37.091377 15380 raft_consensus.cc:3037] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:37.091732 15589 raft_consensus.cc:348] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.091995 15589 raft_consensus.cc:374] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:37.092087 15589 raft_consensus.cc:729] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421c91a0052d402da4d53553650fff6c, State: Initialized, Role: FOLLOWER
I20260924 10:01:37.092772 15612 tablet_replica.cc:330] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: stopping tablet replica
I20260924 10:01:37.093297 15612 raft_consensus.cc:2229] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:37.093825 15612 raft_consensus.cc:2258] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:37.089066 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470e8e9ea34e428facbf7e477a9ec13d" candidate_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421c91a0052d402da4d53553650fff6c"
I20260924 10:01:37.094290 15589 consensus_queue.cc:260] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.094648 15518 raft_consensus.cc:3037] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:37.095024 15589 ts_tablet_manager.cc:1428] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 10:01:37.095775 15613 tablet_replica.cc:330] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: stopping tablet replica
I20260924 10:01:37.095989 15611 tablet_replica.cc:330] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: stopping tablet replica
I20260924 10:01:37.096197 15613 raft_consensus.cc:2229] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:37.096333 15611 raft_consensus.cc:2229] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:01:37.096659 15613 raft_consensus.cc:2258] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:37.097003 15611 raft_consensus.cc:2258] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:37.084410 15519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470e8e9ea34e428facbf7e477a9ec13d" candidate_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421c91a0052d402da4d53553650fff6c" is_pre_election: true
I20260924 10:01:37.097601 15380 raft_consensus.cc:2454] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9ae7c7eef8401ebd0f19b45106f29d in term 1.
I20260924 10:01:37.098258 15171 leader_election.cc:302] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9ae7c7eef8401ebd0f19b45106f29d, 817571f429ab4aafb7d2cebfc141187c; no voters:
I20260924 10:01:37.098620 15594 raft_consensus.cc:2781] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:37.099457 15518 raft_consensus.cc:2454] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9ae7c7eef8401ebd0f19b45106f29d in term 1.
I20260924 10:01:37.099613 15612 ts_tablet_manager.cc:1905] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:37.100915 15613 ts_tablet_manager.cc:1905] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:37.106047 15612 ts_tablet_manager.cc:1918] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:01:37.106264 15612 log.cc:1196] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/27365fb9625f408b946004be68ce0765
I20260924 10:01:37.106813 15612 ts_tablet_manager.cc:1939] T 27365fb9625f408b946004be68ce0765 P 817571f429ab4aafb7d2cebfc141187c: Deleting consensus metadata
I20260924 10:01:37.107654 15613 ts_tablet_manager.cc:1918] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:01:37.107676 15611 ts_tablet_manager.cc:1905] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:37.107928 15613 log.cc:1196] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/27365fb9625f408b946004be68ce0765
I20260924 10:01:37.108445 15613 ts_tablet_manager.cc:1939] T 27365fb9625f408b946004be68ce0765 P 6a9ae7c7eef8401ebd0f19b45106f29d: Deleting consensus metadata
I20260924 10:01:37.108701 15068 catalog_manager.cc:4841] TS 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811): tablet 27365fb9625f408b946004be68ce0765 (table test-table [id=5c0840b6c57c4e40aa282f06591e05d3]) successfully deleted
I20260924 10:01:37.110230 15068 catalog_manager.cc:4841] TS 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043): tablet 27365fb9625f408b946004be68ce0765 (table test-table [id=5c0840b6c57c4e40aa282f06591e05d3]) successfully deleted
I20260924 10:01:37.110404 15594 raft_consensus.cc:686] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 LEADER]: Becoming Leader. State: Replica: 6a9ae7c7eef8401ebd0f19b45106f29d, State: Running, Role: LEADER
I20260924 10:01:37.110824 15594 consensus_queue.cc:237] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } }
I20260924 10:01:37.115747 15611 ts_tablet_manager.cc:1918] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:01:37.115996 15611 log.cc:1196] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/27365fb9625f408b946004be68ce0765
I20260924 10:01:37.116539 15611 ts_tablet_manager.cc:1939] T 27365fb9625f408b946004be68ce0765 P 421c91a0052d402da4d53553650fff6c: Deleting consensus metadata
I20260924 10:01:37.116389 15083 catalog_manager.cc:5495] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1). New cstate: current_term: 1 leader_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } health_report { overall_health: UNKNOWN } } }
I20260924 10:01:37.118286 15071 catalog_manager.cc:4841] TS 421c91a0052d402da4d53553650fff6c (127.13.162.3:41881): tablet 27365fb9625f408b946004be68ce0765 (table test-table [id=5c0840b6c57c4e40aa282f06591e05d3]) successfully deleted
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244095, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:37.167132 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.152890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.13.162.62:36027)
0924 10:01:37.153072 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.153082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.153632 (+ 550us) server_negotiation.cc:407] Connection header received
0924 10:01:37.153766 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.153772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.153874 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.153985 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:37.155095 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.156214 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.157393 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.158060 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.158181 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.161710 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:37.161715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:37.161718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:37.161745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:37.164483 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.165331 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.165335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.165337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.165467 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.165837 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.165840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.165842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.166099 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:37.166198 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.166703 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.166875 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":63}
I20260924 10:01:37.167565 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.152943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36027 (local address 127.0.0.1:57992)
0924 10:01:37.153535 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:37.153550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:37.153667 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:37.154015 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:37.154018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:37.154702 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:37.154959 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.154967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.156329 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.156332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.157266 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.157276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.158213 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.158217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.158361 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.158951 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:37.158972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:37.161547 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:37.164602 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:37.164614 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:37.164622 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:37.165228 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:37.165563 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:37.165568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:37.165572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:37.165742 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:37.166212 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:37.166220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:37.166450 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 10:01:37.166630 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 10:01:37.167033 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":99,"thread_start_us":294,"threads_started":1}
I20260924 10:01:37.172564 15083 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:01:37.173600 15083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:37.180356 15360 tablet_service.cc:1467] Processing CreateTablet for tablet 9f593dbd43fa4837b7609f3453e6e44c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bbe799bb0fd45aab0eb848a2f579e91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:01:37.180900 15360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f593dbd43fa4837b7609f3453e6e44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:37.186437 15221 tablet_service.cc:1467] Processing CreateTablet for tablet 9f593dbd43fa4837b7609f3453e6e44c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bbe799bb0fd45aab0eb848a2f579e91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:01:37.186982 15221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f593dbd43fa4837b7609f3453e6e44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:37.187124 15499 tablet_service.cc:1467] Processing CreateTablet for tablet 9f593dbd43fa4837b7609f3453e6e44c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bbe799bb0fd45aab0eb848a2f579e91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:01:37.187628 15499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f593dbd43fa4837b7609f3453e6e44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:37.190555 15588 tablet_bootstrap.cc:492] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: Bootstrap starting.
I20260924 10:01:37.192142 15588 tablet_bootstrap.cc:654] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:37.193630 15589 tablet_bootstrap.cc:492] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: Bootstrap starting.
I20260924 10:01:37.194970 15590 tablet_bootstrap.cc:492] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: Bootstrap starting.
I20260924 10:01:37.195174 15589 tablet_bootstrap.cc:654] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:37.196486 15590 tablet_bootstrap.cc:654] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:37.201869 15588 tablet_bootstrap.cc:492] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: No bootstrap required, opened a new log
I20260924 10:01:37.202019 15588 ts_tablet_manager.cc:1397] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20260924 10:01:37.202740 15588 raft_consensus.cc:348] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.202987 15588 raft_consensus.cc:374] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:37.203056 15588 raft_consensus.cc:729] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817571f429ab4aafb7d2cebfc141187c, State: Initialized, Role: FOLLOWER
I20260924 10:01:37.203275 15588 consensus_queue.cc:260] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.203896 15595 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.204219 15595 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.205338 15588 ts_tablet_manager.cc:1428] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:37.208788 15590 tablet_bootstrap.cc:492] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: No bootstrap required, opened a new log
I20260924 10:01:37.209009 15590 ts_tablet_manager.cc:1397] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260924 10:01:37.209724 15589 tablet_bootstrap.cc:492] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: No bootstrap required, opened a new log
I20260924 10:01:37.209823 15590 raft_consensus.cc:348] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.209949 15589 ts_tablet_manager.cc:1397] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.016s
I20260924 10:01:37.210104 15590 raft_consensus.cc:374] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:37.210222 15590 raft_consensus.cc:729] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9ae7c7eef8401ebd0f19b45106f29d, State: Initialized, Role: FOLLOWER
I20260924 10:01:37.210618 15590 consensus_queue.cc:260] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.210712 15589 raft_consensus.cc:348] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.210949 15589 raft_consensus.cc:374] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:37.211052 15589 raft_consensus.cc:729] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421c91a0052d402da4d53553650fff6c, State: Initialized, Role: FOLLOWER
I20260924 10:01:37.211186 15594 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.211452 15594 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.211328 15589 consensus_queue.cc:260] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.211702 15598 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.211997 15598 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.212423 15590 ts_tablet_manager.cc:1428] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 10:01:37.213197 15589 ts_tablet_manager.cc:1428] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260924 10:01:37.239292 15570 tablet.cc:2369] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:01:37.260324 15431 tablet.cc:2369] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:01:37.265584 15292 tablet.cc:2369] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:01:37.328843 15598 raft_consensus.cc:482] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:37.329099 15598 raft_consensus.cc:504] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.329701 15598 leader_election.cc:288] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043), 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811)
I20260924 10:01:37.330360 15380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f593dbd43fa4837b7609f3453e6e44c" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817571f429ab4aafb7d2cebfc141187c" is_pre_election: true
I20260924 10:01:37.330374 15241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f593dbd43fa4837b7609f3453e6e44c" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" is_pre_election: true
I20260924 10:01:37.330595 15380 raft_consensus.cc:2454] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 0.
I20260924 10:01:37.330650 15241 raft_consensus.cc:2454] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 0.
I20260924 10:01:37.331080 15450 leader_election.cc:302] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 421c91a0052d402da4d53553650fff6c, 6a9ae7c7eef8401ebd0f19b45106f29d; no voters:
I20260924 10:01:37.331324 15598 raft_consensus.cc:2781] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:37.331432 15598 raft_consensus.cc:482] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:37.331543 15598 raft_consensus.cc:3037] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:37.333832 15598 raft_consensus.cc:504] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.334426 15598 leader_election.cc:288] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 election: Requested vote from peers 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043), 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811)
I20260924 10:01:37.334895 15241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f593dbd43fa4837b7609f3453e6e44c" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d"
I20260924 10:01:37.334959 15380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f593dbd43fa4837b7609f3453e6e44c" candidate_uuid: "421c91a0052d402da4d53553650fff6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817571f429ab4aafb7d2cebfc141187c"
I20260924 10:01:37.335071 15241 raft_consensus.cc:3037] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:37.335214 15380 raft_consensus.cc:3037] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:37.337046 15241 raft_consensus.cc:2454] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 1.
I20260924 10:01:37.337404 15450 leader_election.cc:302] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 421c91a0052d402da4d53553650fff6c, 6a9ae7c7eef8401ebd0f19b45106f29d; no voters:
I20260924 10:01:37.337630 15598 raft_consensus.cc:2781] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:37.337800 15598 raft_consensus.cc:686] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [term 1 LEADER]: Becoming Leader. State: Replica: 421c91a0052d402da4d53553650fff6c, State: Running, Role: LEADER
I20260924 10:01:37.338130 15380 raft_consensus.cc:2454] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421c91a0052d402da4d53553650fff6c in term 1.
I20260924 10:01:37.338249 15598 consensus_queue.cc:237] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } }
I20260924 10:01:37.345979 15596 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: TxnStatusTablet state changed. Reason: New leader 421c91a0052d402da4d53553650fff6c. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.346213 15596 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:01:37.346947 15631 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:01:37.347227 15631 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:01:37.348202 15083 catalog_manager.cc:5495] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 421c91a0052d402da4d53553650fff6c (127.13.162.3). New cstate: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } health_report { overall_health: UNKNOWN } } }
I20260924 10:01:37.397588 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.392445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41881 (local address 127.0.0.1:39268)
0924 10:01:37.392635 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:37.392654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:37.392761 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:37.393154 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:37.393157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:37.393166 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:37.393312 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.393319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.394650 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.394654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.395623 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.395633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.396587 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.396591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.396724 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.396789 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.397194 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.397204 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:01:37.397254 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 10:01:37.397356 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:01:37.397665 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.392542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.13.162.3:41881)
0924 10:01:37.392724 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.392729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.392750 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:01:37.392936 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.392942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.393034 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.393131 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:37.393424 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.394539 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.395763 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.396458 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.396589 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.396966 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.397104 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.397193 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.397298 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.397375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:01:37.401085 15241 raft_consensus.cc:1261] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Refusing update from remote peer 421c91a0052d402da4d53553650fff6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:37.401134 15380 raft_consensus.cc:1261] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Refusing update from remote peer 421c91a0052d402da4d53553650fff6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:37.401706 15596 consensus_queue.cc:1035] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:37.402063 15598 consensus_queue.cc:1035] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:37.406234 15595 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: TxnStatusTablet state changed. Reason: New leader 421c91a0052d402da4d53553650fff6c. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.406541 15595 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.407490 15596 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.407729 15596 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:01:37.411857 15595 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.412061 15595 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 817571f429ab4aafb7d2cebfc141187c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.415216 15594 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: TxnStatusTablet state changed. Reason: New leader 421c91a0052d402da4d53553650fff6c. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.415479 15594 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.417481 15598 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.417768 15598 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 421c91a0052d402da4d53553650fff6c: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:01:37.418963 15594 tablet_replica.cc:439] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "421c91a0052d402da4d53553650fff6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9ae7c7eef8401ebd0f19b45106f29d" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42043 } } peers { permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 } } peers { permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 } } }
I20260924 10:01:37.419219 15594 tablet_replica.cc:442] T 9f593dbd43fa4837b7609f3453e6e44c P 6a9ae7c7eef8401ebd0f19b45106f29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:37.423048 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.414480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.13.162.1:42043)
0924 10:01:37.416257 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.416263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.416280 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:01:37.416371 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.416376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.416471 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.417164 (+ 693us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:37.417330 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.418548 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.420401 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.420922 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.421049 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.421648 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.422410 (+ 762us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.422563 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.422620 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.422700 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1659,"thread_start_us":1543,"threads_started":1}
I20260924 10:01:37.438599 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.432692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41881 (local address 127.13.162.1:33287)
0924 10:01:37.433278 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:37.433297 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:37.433407 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:37.433747 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:37.433752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:37.433767 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:37.434078 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.434087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.435470 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.435473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.436537 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.436548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.437556 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.437560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.437681 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.437730 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.438148 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.438154 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:01:37.438204 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 10:01:37.438298 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
I20260924 10:01:37.438627 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.432732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:33287 (local address 127.13.162.3:41881)
0924 10:01:37.432915 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.432922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.433365 (+ 443us) server_negotiation.cc:407] Connection header received
0924 10:01:37.433509 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.433515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.433594 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.433680 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:37.434209 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.435354 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.436667 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.437400 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.437515 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.437897 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.438061 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.438155 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.438242 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.438338 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:01:37.447992 15380 raft_consensus.cc:1261] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Refusing update from remote peer 6a9ae7c7eef8401ebd0f19b45106f29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:37.448007 15518 raft_consensus.cc:1261] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Refusing update from remote peer 6a9ae7c7eef8401ebd0f19b45106f29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:37.448693 15594 consensus_queue.cc:1035] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "421c91a0052d402da4d53553650fff6c" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:37.449287 15594 consensus_queue.cc:1035] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "817571f429ab4aafb7d2cebfc141187c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:37.488152 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.480421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:42043 (local address 127.13.162.3:60191)
0924 10:01:37.482811 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:37.482827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:37.482947 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:37.483238 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:37.483241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:37.483250 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:37.483512 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.483519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.484848 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.484853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.486201 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:37.486213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.487132 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.487139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:37.487259 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.487318 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.487689 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.487698 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:01:37.487755 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 10:01:37.487860 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"thread_start_us":2078,"threads_started":1}
I20260924 10:01:37.488173 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.480811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:60191 (local address 127.13.162.1:42043)
0924 10:01:37.481029 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.481035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.482907 (+ 1872us) server_negotiation.cc:407] Connection header received
0924 10:01:37.483059 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.483065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.483143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.483222 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:37.483633 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.484711 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.486362 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.486955 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.487081 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.487487 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:37.487609 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:37.487694 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.487792 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.487870 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 10:01:37.526808 15082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:37.527109 15082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:37.530553 15082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9902316c6a7740b9b346762e698b5517: Sending DeleteTablet for 3 replicas of tablet 470e8e9ea34e428facbf7e477a9ec13d
I20260924 10:01:37.531395 15221 tablet_service.cc:1514] Processing DeleteTablet for tablet 470e8e9ea34e428facbf7e477a9ec13d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:37 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:47930
I20260924 10:01:37.531716 15613 tablet_replica.cc:330] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: stopping tablet replica
I20260924 10:01:37.531913 15613 raft_consensus.cc:2229] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:01:37.531987 15360 tablet_service.cc:1514] Processing DeleteTablet for tablet 470e8e9ea34e428facbf7e477a9ec13d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:37 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57740
I20260924 10:01:37.532405 15613 raft_consensus.cc:2258] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:37.532688 15612 tablet_replica.cc:330] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: stopping tablet replica
I20260924 10:01:37.532867 15612 raft_consensus.cc:2229] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:37.533034 15612 raft_consensus.cc:2258] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:37.533645 15499 tablet_service.cc:1514] Processing DeleteTablet for tablet 470e8e9ea34e428facbf7e477a9ec13d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:37 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:39254
I20260924 10:01:37.533946 15611 tablet_replica.cc:330] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: stopping tablet replica
I20260924 10:01:37.534149 15611 raft_consensus.cc:2229] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:37.534371 15611 raft_consensus.cc:2258] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:37.534483 15612 ts_tablet_manager.cc:1905] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:37.535434 15611 ts_tablet_manager.cc:1905] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:37.539450 15613 ts_tablet_manager.cc:1905] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:37.545280 15611 ts_tablet_manager.cc:1918] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:37.545445 15611 log.cc:1196] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/470e8e9ea34e428facbf7e477a9ec13d
I20260924 10:01:37.545969 15611 ts_tablet_manager.cc:1939] T 470e8e9ea34e428facbf7e477a9ec13d P 421c91a0052d402da4d53553650fff6c: Deleting consensus metadata
I20260924 10:01:37.548916 15071 catalog_manager.cc:4841] TS 421c91a0052d402da4d53553650fff6c (127.13.162.3:41881): tablet 470e8e9ea34e428facbf7e477a9ec13d (table test-table [id=3a542828334349cf982030e6fca6eca3]) successfully deleted
I20260924 10:01:37.549934 15612 ts_tablet_manager.cc:1918] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:37.550431 15612 log.cc:1196] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/470e8e9ea34e428facbf7e477a9ec13d
I20260924 10:01:37.550896 15612 ts_tablet_manager.cc:1939] T 470e8e9ea34e428facbf7e477a9ec13d P 817571f429ab4aafb7d2cebfc141187c: Deleting consensus metadata
I20260924 10:01:37.551113 15613 ts_tablet_manager.cc:1918] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:37.551301 15613 log.cc:1196] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/470e8e9ea34e428facbf7e477a9ec13d
I20260924 10:01:37.552392 15068 catalog_manager.cc:4841] TS 817571f429ab4aafb7d2cebfc141187c (127.13.162.2:41811): tablet 470e8e9ea34e428facbf7e477a9ec13d (table test-table [id=3a542828334349cf982030e6fca6eca3]) successfully deleted
I20260924 10:01:37.552996 15613 ts_tablet_manager.cc:1939] T 470e8e9ea34e428facbf7e477a9ec13d P 6a9ae7c7eef8401ebd0f19b45106f29d: Deleting consensus metadata
I20260924 10:01:37.554054 15068 catalog_manager.cc:4841] TS 6a9ae7c7eef8401ebd0f19b45106f29d (127.13.162.1:42043): tablet 470e8e9ea34e428facbf7e477a9ec13d (table test-table [id=3a542828334349cf982030e6fca6eca3]) successfully deleted
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:37.574818 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.555524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.13.162.1:42043)
0924 10:01:37.555706 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.555712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.556360 (+ 648us) server_negotiation.cc:407] Connection header received
0924 10:01:37.556544 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.556549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.556631 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.556751 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:37.558659 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.559771 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.561602 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.562269 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.562440 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.567333 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:37.567352 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:37.567371 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:37.567412 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:37.571592 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.572379 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.572412 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.572416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.572562 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.573059 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.573065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.573069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.573572 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:37.573954 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.574143 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.574408 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 10:01:37.575538 15286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47942
I20260924 10:01:37.599102 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.579232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.13.162.62:36027)
0924 10:01:37.579430 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.579436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.584414 (+ 4978us) server_negotiation.cc:407] Connection header received
0924 10:01:37.584587 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.584593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.584676 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.584787 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:37.586409 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.587531 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.589349 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.590006 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.590148 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.592962 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:37.592969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:37.592974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:37.593013 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:37.596327 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.597074 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.597079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.597083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.597224 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.597596 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.597601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.597604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.597940 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:37.598220 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.598518 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.598735 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:01:37.599884 15082 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57994
I20260924 10:01:37.602077 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15155
I20260924 10:01:37.619728 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.604310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.13.162.1:42043)
0924 10:01:37.604493 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.604499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.604899 (+ 400us) server_negotiation.cc:407] Connection header received
0924 10:01:37.605033 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.605038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.605132 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.605235 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:37.606619 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.607770 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.609505 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.610238 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.610482 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.613180 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:37.613187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:37.613192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:37.613230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:37.616811 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.617606 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.617612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.617616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.617750 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.618216 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.618225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.618228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.618609 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:37.618859 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.619147 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.619373 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:01:37.620285 15286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47948
W20260924 10:01:37.620915 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15155: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:37.621101 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15155
I20260924 10:01:37.635780 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15295
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:01:37.652292 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.636569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.13.162.2:41811)
0924 10:01:37.637101 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.637106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.637122 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:01:37.637174 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.637178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.637237 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.637327 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:37.638854 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.639562 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.641331 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.641751 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.641905 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.646590 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:37.646605 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:37.646641 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:37.646680 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:37.649675 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.650493 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.650501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.650503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.650632 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.650987 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.650990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.650992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.651312 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:37.651515 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.651824 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.651981 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":310,"threads_started":1}
W20260924 10:01:37.652949 15425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57752
W20260924 10:01:37.653558 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15295: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:37.653731 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15295
I20260924 10:01:37.667138 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15434
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:01:37.682895 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:37.667888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.13.162.3:41881)
0924 10:01:37.668104 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:37.668109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:37.668128 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:01:37.668258 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:37.668264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:37.668347 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:37.668461 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:37.669625 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.670787 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.672023 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:37.672715 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:37.672841 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:37.677268 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:37.677290 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:37.677311 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:37.677359 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:37.680129 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.680905 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.680912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.680914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.681035 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:37.681369 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:37.681372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:37.681374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:37.681717 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:37.681955 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:37.682224 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 10:01:37.682566 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 10:01:37.683591 15564 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39280
W20260924 10:01:37.684202 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15434: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:37.684348 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15434
I20260924 10:01:37.701617 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15052
I20260924 10:01:37.702695 15124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:37.812930 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15052
2026-09-24T10:01:37Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4181 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15656](info): set up 2 sockets
Sep 24 10:01:37 dist-test-slave-jvsj krb5kdc[15656](info): commencing operation
krb5kdc: starting...
W20260924 10:01:38.794154 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:38 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:38.958129 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:39197
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:39197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:39.263439 15669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:39.263721 15669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:39.263844 15669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:39.271087 15669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:39.271195 15669 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:01:39.271353 15669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:39.271485 15669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:39.271551 15669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:39.271637 15669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:39.281699 15669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:39197
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:39.283042 15669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:39.284324 15669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:39.298928 15674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:39.298897 15677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:39.298861 15675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:39.299587 15669 server_base.cc:1034] running on GCE node
I20260924 10:01:39.300498 15669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:39.301949 15669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:39.303205 15669 hybrid_clock.cc:648] HybridClock initialized: now 1790244099303137 us; error 40 us; skew 500 ppm
I20260924 10:01:39.303561 15669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:39 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:39.310070 15669 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:39.311661 15669 webserver.cc:460] Webserver started at http://127.13.162.62:39641/ using document root <none> and password file <none>
I20260924 10:01:39.312356 15669 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:39.312466 15669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:39.312768 15669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:39.315889 15669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "62eb140e4790414ea03311ce38309820"
format_stamp: "Formatted at 2026-09-24 10:01:39 on dist-test-slave-jvsj"
server_key: "6eab8a5d6faec7f6c127bb4e368ee535"
server_key_iv: "94bfde705d037c3c912b7591d0098958"
server_key_version: "encryptionkey@0"
I20260924 10:01:39.316634 15669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "62eb140e4790414ea03311ce38309820"
format_stamp: "Formatted at 2026-09-24 10:01:39 on dist-test-slave-jvsj"
server_key: "6eab8a5d6faec7f6c127bb4e368ee535"
server_key_iv: "94bfde705d037c3c912b7591d0098958"
server_key_version: "encryptionkey@0"
I20260924 10:01:39.321772 15669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 10:01:39.325188 15684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:39.326588 15669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:39.326745 15669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "62eb140e4790414ea03311ce38309820"
format_stamp: "Formatted at 2026-09-24 10:01:39 on dist-test-slave-jvsj"
server_key: "6eab8a5d6faec7f6c127bb4e368ee535"
server_key_iv: "94bfde705d037c3c912b7591d0098958"
server_key_version: "encryptionkey@0"
I20260924 10:01:39.326933 15669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:39.354143 15669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:39.358062 15669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:39.358305 15669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:39.377997 15669 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:39197
I20260924 10:01:39.378026 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:39197 every 8 connection(s)
I20260924 10:01:39.379312 15669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:39.383033 15746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:39.383623 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15669
I20260924 10:01:39.383792 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:39.384382 13960 external_mini_cluster.cc:1428] Setting key 4481a0774584eddceb0d91641ca4cf1f
I20260924 10:01:39.396672 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Bootstrap starting.
Sep 24 10:01:39 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:39.399952 15746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:39.401261 15746 log.cc:824] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:39.404377 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: No bootstrap required, opened a new log
I20260924 10:01:39.407188 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:39.386279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.13.162.62:39197)
0924 10:01:39.387058 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:39.387077 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:39.387133 (+ 56us) server_negotiation.cc:407] Connection header received
0924 10:01:39.389106 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:39.389134 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:39.389667 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:39.390102 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:39.391375 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.393398 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.394684 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.395754 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.395954 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:39.399761 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:39.399772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:39.399791 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:39.399841 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:39.403335 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:39.403959 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:39.403966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:39.403973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:39.404175 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:39.404527 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:39.404531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:39.404533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:39.405056 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:39.405317 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:39.405758 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 10:01:39.406154 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":283,"threads_started":1}
I20260924 10:01:39.412343 15746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62eb140e4790414ea03311ce38309820" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39197 } }
I20260924 10:01:39.412690 15746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:39.412777 15746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62eb140e4790414ea03311ce38309820, State: Initialized, Role: FOLLOWER
I20260924 10:01:39.413288 15746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62eb140e4790414ea03311ce38309820" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39197 } }
I20260924 10:01:39.413497 15746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:39.413615 15746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:39.413759 15746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:39.415866 15746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62eb140e4790414ea03311ce38309820" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39197 } }
I20260924 10:01:39.416325 15746 leader_election.cc:302] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62eb140e4790414ea03311ce38309820; no voters:
I20260924 10:01:39.416929 15746 leader_election.cc:288] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:39.417042 15751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:39.417697 15751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [term 1 LEADER]: Becoming Leader. State: Replica: 62eb140e4790414ea03311ce38309820, State: Running, Role: LEADER
I20260924 10:01:39.418246 15751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62eb140e4790414ea03311ce38309820" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39197 } }
I20260924 10:01:39.419301 15746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:39.421136 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62eb140e4790414ea03311ce38309820. Latest consensus state: current_term: 1 leader_uuid: "62eb140e4790414ea03311ce38309820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62eb140e4790414ea03311ce38309820" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39197 } } }
I20260924 10:01:39.421182 15752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62eb140e4790414ea03311ce38309820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62eb140e4790414ea03311ce38309820" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39197 } } }
I20260924 10:01:39.421522 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:39.421528 15752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820 [sys.catalog]: This master's current role is: LEADER
W20260924 10:01:39.434607 15767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:01:39.434749 15767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:01:39.434880 15757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:39.436848 15757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:39.444957 15757 catalog_manager.cc:1347] Generated new cluster ID: 89dbcf47a72040c787f64ef943567453
I20260924 10:01:39.445067 15757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:39.476995 15757 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:39.478305 15757 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:39.484431 15757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Generated new TSK 0
I20260924 10:01:39.485445 15757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:39.538249 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39197
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:39.849351 15772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:39.849628 15772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:39.849762 15772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:39.856952 15772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:39.857067 15772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:39.857184 15772 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:01:39.857244 15772 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:01:39.857342 15772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:39.867045 15772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:39.868367 15772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:39.869622 15772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:39.881629 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:39.881738 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:39.881781 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:39.882609 15772 server_base.cc:1034] running on GCE node
I20260924 10:01:39.883160 15772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:39.884038 15772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:39.885262 15772 hybrid_clock.cc:648] HybridClock initialized: now 1790244099885206 us; error 43 us; skew 500 ppm
I20260924 10:01:39.885584 15772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:39 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:39.888630 15772 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:39.890034 15772 webserver.cc:460] Webserver started at http://127.13.162.1:39353/ using document root <none> and password file <none>
I20260924 10:01:39.890769 15772 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:39.890909 15772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:39.891212 15772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:39.894130 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "75154b9842f147a3bb0018ba890cc0be"
format_stamp: "Formatted at 2026-09-24 10:01:39 on dist-test-slave-jvsj"
server_key: "2e38faf43f7f74717bb9fb1f02370045"
server_key_iv: "52ff5bd7f69572638bf343c9b36960fd"
server_key_version: "encryptionkey@0"
I20260924 10:01:39.894881 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "75154b9842f147a3bb0018ba890cc0be"
format_stamp: "Formatted at 2026-09-24 10:01:39 on dist-test-slave-jvsj"
server_key: "2e38faf43f7f74717bb9fb1f02370045"
server_key_iv: "52ff5bd7f69572638bf343c9b36960fd"
server_key_version: "encryptionkey@0"
I20260924 10:01:39.899873 15772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:01:39.903152 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:39.904564 15772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:39.904713 15772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "75154b9842f147a3bb0018ba890cc0be"
format_stamp: "Formatted at 2026-09-24 10:01:39 on dist-test-slave-jvsj"
server_key: "2e38faf43f7f74717bb9fb1f02370045"
server_key_iv: "52ff5bd7f69572638bf343c9b36960fd"
server_key_version: "encryptionkey@0"
I20260924 10:01:39.904896 15772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:39.923046 15772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:39.927178 15772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:39.927407 15772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:39.930017 15772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:39.930140 15772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:39.930296 15772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:39.930430 15772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:01:39 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:39.949847 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:39.930419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:44647 (local address 127.13.162.62:39197)
0924 10:01:39.932574 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:39.932581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:39.932602 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:01:39.933963 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:39.933969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:39.934048 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:39.934141 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:39.935712 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.936417 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.938858 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.939522 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.939662 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:39.943796 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:39.943803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:39.943809 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:39.943853 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:39.947019 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:39.947680 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:39.947686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:39.947689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:39.947808 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:39.948204 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:39.948209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:39.948213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:39.948479 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:39.948600 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:39.949296 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 10:01:39.949509 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2039,"thread_start_us":1944,"threads_started":1}
I20260924 10:01:39.951226 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:39.930999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.13.162.1:44647)
0924 10:01:39.931826 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:39.931887 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:39.933705 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:39.934431 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:39.934440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:39.934958 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:39.935555 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:39.935575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.936560 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.936565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:39.938703 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:39.938717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.939766 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.939774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:39.940051 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:39.940833 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:39.940867 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:39.943537 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:39.947149 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:39.947159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:39.947179 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:39.947563 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:39.947907 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:39.947913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:39.947917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:39.948096 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:39.948592 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:39.948599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:39.948945 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 10:01:39.949161 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 10:01:39.949671 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":385,"threads_started":1}
I20260924 10:01:39.969928 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:39.956922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.13.162.1:42309)
0924 10:01:39.957461 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:39.957476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:39.957598 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:39.957903 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:39.957909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:39.958544 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:39.959002 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:39.959016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.959969 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.959975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:39.961579 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:39.961595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.962397 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.962404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:39.962616 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:39.963276 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:39.963299 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:39.964272 (+ 973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:39.967426 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:39.967435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:39.967440 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:39.967965 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:39.968302 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:39.968310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:39.968314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:39.968484 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:39.969096 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:39.969101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:39.969277 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 10:01:39.969386 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:01:39.969602 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
I20260924 10:01:39.970281 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:39.956834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:42309 (local address 127.13.162.62:39197)
0924 10:01:39.957003 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:39.957007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:39.957550 (+ 543us) server_negotiation.cc:407] Connection header received
0924 10:01:39.957713 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:39.957718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:39.957775 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:39.957859 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:39.959153 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.959848 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.961710 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:39.962189 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:39.962306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:39.964511 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:39.964517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:39.964522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:39.964559 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:39.967298 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:39.968071 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:39.968074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:39.968077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:39.968191 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:39.968600 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:39.968605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:39.968609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:39.968966 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:39.969111 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:39.969497 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 10:01:39.969741 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:01:39.979739 15772 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:33959
I20260924 10:01:39.979784 15907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:33959 every 8 connection(s)
I20260924 10:01:39.980903 15772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:39.982703 15908 heartbeater.cc:347] Connected to a master server at 127.13.162.62:39197
I20260924 10:01:39.982909 15908 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:39.983508 15908 heartbeater.cc:511] Master 127.13.162.62:39197 requested a full tablet report, sending...
I20260924 10:01:39.984547 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15772
I20260924 10:01:39.984721 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:39.985307 13960 external_mini_cluster.cc:1428] Setting key 0412d0de15555e5b5193d135281d2a6f
I20260924 10:01:39.985567 15701 ts_manager.cc:194] Registered new tserver with Master: 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959)
I20260924 10:01:39.987242 15701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:44647
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:40.020823 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39197
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:40.323414 15912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:40.323699 15912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:40.323849 15912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:40.331151 15912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:40.331270 15912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:40.331396 15912 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:01:40.331461 15912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:01:40.331562 15912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:40.341071 15912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:40.342371 15912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:40.343652 15912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:40.356199 15918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:40.356160 15920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:40.356217 15917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:40.356958 15912 server_base.cc:1034] running on GCE node
I20260924 10:01:40.357530 15912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:40.358477 15912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:40.359701 15912 hybrid_clock.cc:648] HybridClock initialized: now 1790244100359649 us; error 44 us; skew 500 ppm
I20260924 10:01:40.359998 15912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:40 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:40.362950 15912 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:40.364336 15912 webserver.cc:460] Webserver started at http://127.13.162.2:39991/ using document root <none> and password file <none>
I20260924 10:01:40.364990 15912 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:40.365090 15912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:40.365345 15912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:40.368184 15912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d"
format_stamp: "Formatted at 2026-09-24 10:01:40 on dist-test-slave-jvsj"
server_key: "bb990ee730f97af33df8e42d5a47287b"
server_key_iv: "aadfe73032dd91b1438fdf16f019e091"
server_key_version: "encryptionkey@0"
I20260924 10:01:40.368873 15912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d"
format_stamp: "Formatted at 2026-09-24 10:01:40 on dist-test-slave-jvsj"
server_key: "bb990ee730f97af33df8e42d5a47287b"
server_key_iv: "aadfe73032dd91b1438fdf16f019e091"
server_key_version: "encryptionkey@0"
I20260924 10:01:40.373921 15912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:01:40.377329 15927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:40.378973 15912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:40.379128 15912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d"
format_stamp: "Formatted at 2026-09-24 10:01:40 on dist-test-slave-jvsj"
server_key: "bb990ee730f97af33df8e42d5a47287b"
server_key_iv: "aadfe73032dd91b1438fdf16f019e091"
server_key_version: "encryptionkey@0"
I20260924 10:01:40.379338 15912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:40.402437 15912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:40.406503 15912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:40.406734 15912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:40.409294 15912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:40.409386 15912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:40.409525 15912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:40.409626 15912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:01:40 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:40.434543 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.410390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.13.162.2:40691)
0924 10:01:40.411282 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.411331 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.413153 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.413935 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.413947 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.414736 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:40.415629 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.415659 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.416937 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.416943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.419520 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.419538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.420591 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.420600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.420961 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.421745 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:40.421772 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:40.424746 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:40.428983 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.428993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.429009 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.429580 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.430075 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.430081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.430084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.430397 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.431194 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:40.431201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:40.431741 (+ 540us) client_negotiation.cc:769] Sending connection context
0924 10:01:40.432084 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 10:01:40.432727 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":398,"threads_started":1}
I20260924 10:01:40.435165 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.409811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:40691 (local address 127.13.162.62:39197)
0924 10:01:40.409992 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.409998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.411474 (+ 1476us) server_negotiation.cc:407] Connection header received
0924 10:01:40.413460 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.413466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.413546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.413666 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:40.415859 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.416737 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.419687 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.420314 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.420442 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.425047 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:40.425054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:40.425060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:40.425108 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:40.428817 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.429740 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.429746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.429749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.429919 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.430557 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.430563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.430566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.430938 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:40.431107 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:40.434560 (+ 3453us) server_negotiation.cc:299] Negotiation successful
0924 10:01:40.434813 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:01:40.444597 15912 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:42345
I20260924 10:01:40.444700 16046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:42345 every 8 connection(s)
I20260924 10:01:40.445803 15912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:40.447011 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 15912
I20260924 10:01:40.447181 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:40.447674 13960 external_mini_cluster.cc:1428] Setting key 91b324cd1ad350d917d2ce07706d0251
I20260924 10:01:40.448887 16047 heartbeater.cc:347] Connected to a master server at 127.13.162.62:39197
I20260924 10:01:40.449111 16047 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:40.449855 16047 heartbeater.cc:511] Master 127.13.162.62:39197 requested a full tablet report, sending...
I20260924 10:01:40.451303 15701 ts_manager.cc:194] Registered new tserver with Master: 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345)
I20260924 10:01:40.452445 15701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:40691
I20260924 10:01:40.457072 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.442849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.13.162.2:45711)
0924 10:01:40.443527 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.443544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.443682 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.443973 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.443978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.444580 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:40.444966 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.444979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.445948 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.445954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.447672 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.447686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.448601 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.448607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.448767 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.449422 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:40.449443 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:40.450464 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:40.454462 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.454468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.454472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.454988 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.455429 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.455442 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.455445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.455582 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.456189 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:40.456193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:40.456348 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 10:01:40.456461 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:01:40.456679 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":337,"threads_started":1}
I20260924 10:01:40.457093 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.442590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:45711 (local address 127.13.162.62:39197)
0924 10:01:40.442742 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.442746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.443626 (+ 880us) server_negotiation.cc:407] Connection header received
0924 10:01:40.443797 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.443802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.443867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.443940 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:40.445095 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.445816 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.447843 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.448434 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.448558 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.450641 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:40.450647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:40.450652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:40.450687 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:40.454261 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.455132 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.455137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.455140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.455317 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.455683 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.455688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.455690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.456053 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:40.456190 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:40.456581 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 10:01:40.456813 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:40.489034 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39197
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:40.800117 16051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:40.800400 16051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:40.800537 16051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:40.807632 16051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:40.807745 16051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:40.807874 16051 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:01:40.807941 16051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:01:40.808043 16051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:40.817588 16051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:40.818907 16051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:40.820240 16051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:40.833019 16056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:40.832861 16059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:40.832957 16057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:40.833802 16051 server_base.cc:1034] running on GCE node
I20260924 10:01:40.834374 16051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:40.835223 16051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:40.836409 16051 hybrid_clock.cc:648] HybridClock initialized: now 1790244100836361 us; error 44 us; skew 500 ppm
I20260924 10:01:40.836686 16051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:40 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:40.839643 16051 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:40.841044 16051 webserver.cc:460] Webserver started at http://127.13.162.3:42633/ using document root <none> and password file <none>
I20260924 10:01:40.841672 16051 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:40.841784 16051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:40.842077 16051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:40.845037 16051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "7b97c26408b748c7a730bad7246ed699"
format_stamp: "Formatted at 2026-09-24 10:01:40 on dist-test-slave-jvsj"
server_key: "29a160adb82b85e3c8688b57c3cca696"
server_key_iv: "90fa3d12f6e0ffb76036779499ee5ff7"
server_key_version: "encryptionkey@0"
I20260924 10:01:40.845754 16051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "7b97c26408b748c7a730bad7246ed699"
format_stamp: "Formatted at 2026-09-24 10:01:40 on dist-test-slave-jvsj"
server_key: "29a160adb82b85e3c8688b57c3cca696"
server_key_iv: "90fa3d12f6e0ffb76036779499ee5ff7"
server_key_version: "encryptionkey@0"
I20260924 10:01:40.850781 16051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:01:40.854094 16066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:40.855482 16051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:40.855621 16051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "7b97c26408b748c7a730bad7246ed699"
format_stamp: "Formatted at 2026-09-24 10:01:40 on dist-test-slave-jvsj"
server_key: "29a160adb82b85e3c8688b57c3cca696"
server_key_iv: "90fa3d12f6e0ffb76036779499ee5ff7"
server_key_version: "encryptionkey@0"
I20260924 10:01:40.855785 16051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:40.871553 16051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:40.875870 16051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:40.876121 16051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:40.878736 16051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:40.878835 16051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:40.878978 16051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:40.879078 16051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:01:40 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:40.902922 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.879655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.13.162.3:40773)
0924 10:01:40.880446 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.880495 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.882397 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.883160 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.883171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.883910 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:40.884779 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.884801 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.886296 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.886302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.888238 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.888250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.889209 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.889214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.889419 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.890130 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:40.890167 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:40.893400 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:40.897161 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.897170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.897190 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.897730 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.898148 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.898157 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.898161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.898558 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.899388 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:40.899400 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:40.899929 (+ 529us) client_negotiation.cc:769] Sending connection context
0924 10:01:40.900293 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 10:01:40.900956 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":313,"threads_started":1}
I20260924 10:01:40.903091 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.879162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:40773 (local address 127.13.162.62:39197)
0924 10:01:40.879379 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.879386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.880610 (+ 1224us) server_negotiation.cc:407] Connection header received
0924 10:01:40.882707 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.882714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.882813 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.882908 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:40.885030 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.886136 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.888372 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.889026 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.889158 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.893630 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:40.893636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:40.893641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:40.893684 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:40.897020 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.897883 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.897887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.897889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.898018 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.898735 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.898740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.898743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.899129 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:40.899308 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:40.902531 (+ 3223us) server_negotiation.cc:299] Negotiation successful
0924 10:01:40.902758 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:01:40.917824 16051 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:45049
I20260924 10:01:40.917925 16185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:45049 every 8 connection(s)
I20260924 10:01:40.919143 16051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:40.921300 16186 heartbeater.cc:347] Connected to a master server at 127.13.162.62:39197
I20260924 10:01:40.921551 16186 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:40.922387 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.909220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.13.162.3:41395)
0924 10:01:40.909839 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.909854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.909988 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.910262 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.910267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.910794 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:40.911222 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.911233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.912468 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.912474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.914158 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.914172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.915265 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.915270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.915379 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.915931 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:40.915947 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:40.916618 (+ 671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:40.919713 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.919719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.919723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.920415 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.920793 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:40.920797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:40.920801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:40.920922 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:40.921438 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:40.921443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:40.921617 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 10:01:40.921751 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 10:01:40.921996 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":308,"threads_started":1}
I20260924 10:01:40.922420 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.909098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:41395 (local address 127.13.162.62:39197)
0924 10:01:40.909229 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.909233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.909942 (+ 709us) server_negotiation.cc:407] Connection header received
0924 10:01:40.910086 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.910092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.910154 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.910228 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:40.911407 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.912273 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.914357 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.914940 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.915033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.916803 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:40.916810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:40.916815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:40.916854 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:40.919601 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.920538 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.920543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.920547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.920691 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.921017 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.921022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.921025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.921327 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:40.921468 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:40.921869 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 10:01:40.922086 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:01:40.922408 16186 heartbeater.cc:511] Master 127.13.162.62:39197 requested a full tablet report, sending...
I20260924 10:01:40.923692 15701 ts_manager.cc:194] Registered new tserver with Master: 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:40.924521 15701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:40773
I20260924 10:01:40.924669 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 16051
I20260924 10:01:40.924829 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:40.925328 13960 external_mini_cluster.cc:1428] Setting key 038b4a879201afc9e242a17de9e68cbc
I20260924 10:01:40.928109 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:01:40 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:01:40 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:40.968569 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.956000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.13.162.62:39197)
0924 10:01:40.956203 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.956209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.956459 (+ 250us) server_negotiation.cc:407] Connection header received
0924 10:01:40.956603 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.956607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.956667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.956761 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:40.958131 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.958922 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.960157 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.960570 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.960683 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.964094 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:40.964099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:40.964102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:40.964133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:40.966244 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.966839 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.966843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.966846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.966961 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:40.967281 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:40.967285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:40.967288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:40.967537 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:40.967754 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:40.968040 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:01:40.968246 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:01:40.972283 15701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:40.978739 15701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:40.989768 15908 heartbeater.cc:503] Master 127.13.162.62:39197 was elected leader, sending a full tablet report...
I20260924 10:01:40.999514 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.989848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.13.162.2:42345)
0924 10:01:40.991644 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.991652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.991680 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:01:40.991959 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.991969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.992234 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.992432 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:40.993318 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.994820 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.997328 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.998200 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.998352 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.998567 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:40.998990 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 10:01:40.999193 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1647,"thread_start_us":1516,"threads_started":1}
I20260924 10:01:40.999571 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.989708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42345 (local address 127.0.0.1:42738)
0924 10:01:40.990914 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.990956 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.991115 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.992669 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.992678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.992736 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:40.993089 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.993102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.994973 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.994979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.997153 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.997165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.998475 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.998485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.998706 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:40.998799 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 10:01:40.998934 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:01:40.999129 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":22,"spinlock_wait_cycles":18688,"thread_start_us":5055,"threads_started":1}
I20260924 10:01:41.001881 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.990287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.13.162.1:33959)
0924 10:01:40.994769 (+ 4482us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.994779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.994806 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:01:40.994899 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.994906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.995100 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.995275 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:40.995813 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.997232 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.999610 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.000960 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.001083 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.001274 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.001461 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.001613 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4323,"thread_start_us":4165,"threads_started":1}
I20260924 10:01:41.001905 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.989813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33959 (local address 127.0.0.1:53924)
0924 10:01:40.992603 (+ 2790us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.992620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.992733 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.995285 (+ 2552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.995289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.995311 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:40.995675 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.995688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.997369 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.997375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:40.999432 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.999445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.001141 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.001147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.001294 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.001325 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.001410 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.001533 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2676,"mutex_wait_us":57,"spinlock_wait_cycles":1792,"thread_start_us":750,"threads_started":1}
I20260924 10:01:41.003182 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.992093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:45049 (local address 127.0.0.1:57830)
0924 10:01:40.995029 (+ 2936us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:40.995045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:40.995159 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:40.995872 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:40.995876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:40.995901 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:40.996279 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:40.996291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.998232 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:40.998237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.000530 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.000543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.002476 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.002483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.002640 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.002669 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.002750 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.002861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2821,"mutex_wait_us":35,"thread_start_us":308,"threads_started":1}
I20260924 10:01:41.003278 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:40.992665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.13.162.3:45049)
0924 10:01:40.993300 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:40.993309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:40.995140 (+ 1831us) server_negotiation.cc:407] Connection header received
0924 10:01:40.995354 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:40.995362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:40.995585 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:40.995735 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:40.996526 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:40.998081 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.000697 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.002284 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.002434 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.002645 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.002842 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.002990 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":331,"threads_started":1}
I20260924 10:01:41.003759 15977 tablet_service.cc:1467] Processing CreateTablet for tablet 90c2ae829ef54f67b71c24686c26af0d (DEFAULT_TABLE table=test-table [id=c007e8cdaa314070811f019fa95afdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:41.005214 15977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c2ae829ef54f67b71c24686c26af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.005554 15838 tablet_service.cc:1467] Processing CreateTablet for tablet 90c2ae829ef54f67b71c24686c26af0d (DEFAULT_TABLE table=test-table [id=c007e8cdaa314070811f019fa95afdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:41.006903 15838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c2ae829ef54f67b71c24686c26af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.006789 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 90c2ae829ef54f67b71c24686c26af0d (DEFAULT_TABLE table=test-table [id=c007e8cdaa314070811f019fa95afdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:41.008076 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c2ae829ef54f67b71c24686c26af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.016781 16205 tablet_bootstrap.cc:492] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Bootstrap starting.
I20260924 10:01:41.019917 16205 tablet_bootstrap.cc:654] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.021162 16205 log.cc:824] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:41.021832 16206 tablet_bootstrap.cc:492] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Bootstrap starting.
I20260924 10:01:41.022107 16207 tablet_bootstrap.cc:492] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Bootstrap starting.
I20260924 10:01:41.025074 16207 tablet_bootstrap.cc:654] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.025460 16206 tablet_bootstrap.cc:654] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.026024 16207 log.cc:824] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:41.026773 16206 log.cc:824] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:41.028877 16205 tablet_bootstrap.cc:492] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: No bootstrap required, opened a new log
I20260924 10:01:41.029166 16205 ts_tablet_manager.cc:1397] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260924 10:01:41.029626 16206 tablet_bootstrap.cc:492] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: No bootstrap required, opened a new log
I20260924 10:01:41.029874 16206 ts_tablet_manager.cc:1397] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 10:01:41.032159 16207 tablet_bootstrap.cc:492] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: No bootstrap required, opened a new log
I20260924 10:01:41.032409 16207 ts_tablet_manager.cc:1397] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 10:01:41.039111 16207 raft_consensus.cc:348] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.039482 16207 raft_consensus.cc:374] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.039592 16207 raft_consensus.cc:729] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.039388 16205 raft_consensus.cc:348] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.039860 16205 raft_consensus.cc:374] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.039992 16205 raft_consensus.cc:729] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.039891 16206 raft_consensus.cc:348] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.040297 16206 raft_consensus.cc:374] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.040177 16207 consensus_queue.cc:260] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.040395 16206 raft_consensus.cc:729] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75154b9842f147a3bb0018ba890cc0be, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.040674 16205 consensus_queue.cc:260] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.041363 16186 heartbeater.cc:503] Master 127.13.162.62:39197 was elected leader, sending a full tablet report...
I20260924 10:01:41.042011 16207 ts_tablet_manager.cc:1428] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 10:01:41.042303 16205 ts_tablet_manager.cc:1428] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 10:01:41.041060 16206 consensus_queue.cc:260] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.042888 16047 heartbeater.cc:503] Master 127.13.162.62:39197 was elected leader, sending a full tablet report...
I20260924 10:01:41.044155 16206 ts_tablet_manager.cc:1428] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260924 10:01:41.113703 16211 raft_consensus.cc:482] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:41.114028 16211 raft_consensus.cc:504] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.115841 16211 leader_election.cc:288] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345), 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959)
I20260924 10:01:41.122671 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.116167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46379 (local address 127.13.162.2:42345)
0924 10:01:41.116381 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.116387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.116578 (+ 191us) server_negotiation.cc:407] Connection header received
0924 10:01:41.116748 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.116753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.116840 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.116928 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:41.117359 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.118413 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.120692 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.121859 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.122021 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.122093 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.122305 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.122430 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 10:01:41.122771 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.116194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42345 (local address 127.13.162.3:46379)
0924 10:01:41.116466 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.116481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.116596 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.116972 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.116977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.117011 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:41.117231 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.117241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.118539 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.118543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.120527 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.120541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.121997 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.122001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.122146 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.122184 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.122275 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.122435 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":74,"spinlock_wait_cycles":35200}
I20260924 10:01:41.122859 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.116322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:60961 (local address 127.13.162.1:33959)
0924 10:01:41.116507 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.116513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.117061 (+ 548us) server_negotiation.cc:407] Connection header received
0924 10:01:41.117197 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.117200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.117253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.117354 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:41.117767 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.118703 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.120885 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.122029 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.122162 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.122208 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.122544 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.122658 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:01:41.123191 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.116221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33959 (local address 127.13.162.3:60961)
0924 10:01:41.116956 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.116972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.117090 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.117373 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.117376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.117393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:41.117630 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.117638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.118825 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.118830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.120755 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.120764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.122214 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.122220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.122399 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.122428 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.122506 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.122629 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":386,"threads_started":1}
I20260924 10:01:41.123769 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2ae829ef54f67b71c24686c26af0d" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" is_pre_election: true
I20260924 10:01:41.124001 15863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2ae829ef54f67b71c24686c26af0d" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be" is_pre_election: true
I20260924 10:01:41.124231 15997 raft_consensus.cc:2454] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 0.
I20260924 10:01:41.124475 15863 raft_consensus.cc:2454] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 0.
I20260924 10:01:41.124894 16070 leader_election.cc:302] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b97c26408b748c7a730bad7246ed699, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:41.125259 16211 raft_consensus.cc:2781] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:41.125408 16211 raft_consensus.cc:482] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:41.125507 16211 raft_consensus.cc:3037] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.127645 16211 raft_consensus.cc:504] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.128196 16211 leader_election.cc:288] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 election: Requested vote from peers 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345), 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959)
I20260924 10:01:41.128593 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2ae829ef54f67b71c24686c26af0d" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d"
I20260924 10:01:41.128628 15863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c2ae829ef54f67b71c24686c26af0d" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be"
I20260924 10:01:41.128765 15997 raft_consensus.cc:3037] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.128786 15863 raft_consensus.cc:3037] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.130987 15863 raft_consensus.cc:2454] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 1.
I20260924 10:01:41.131012 15997 raft_consensus.cc:2454] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 1.
I20260924 10:01:41.131328 16070 leader_election.cc:302] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b97c26408b748c7a730bad7246ed699, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:41.131611 16211 raft_consensus.cc:2781] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:41.132036 16211 raft_consensus.cc:686] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 1 LEADER]: Becoming Leader. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Running, Role: LEADER
I20260924 10:01:41.132470 16211 consensus_queue.cc:237] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.136564 15701 catalog_manager.cc:5495] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b97c26408b748c7a730bad7246ed699 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "7b97c26408b748c7a730bad7246ed699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } health_report { overall_health: HEALTHY } } }
W20260924 10:01:41.170991 16187 tablet.cc:2369] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:01:41 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:41.184983 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.172688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.13.162.62:39197)
0924 10:01:41.172845 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.172850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.173342 (+ 492us) server_negotiation.cc:407] Connection header received
0924 10:01:41.173521 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.173526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.173590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.173666 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:41.174793 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.175494 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.176660 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.177086 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.177186 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.180265 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:41.180269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:41.180272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:41.180299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:41.182546 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:41.183144 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:41.183149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:41.183153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:41.183255 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:41.183569 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:41.183574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:41.183577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:41.183834 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:41.183959 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.184389 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.184667 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":51}
I20260924 10:01:41.184989 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.172722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39197 (local address 127.0.0.1:37282)
0924 10:01:41.173243 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.173259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.173415 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.173713 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.173718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.174409 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:41.174656 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.174665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.175606 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.175611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.176533 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.176544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.177249 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.177255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.177394 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.178011 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:41.178033 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:41.180113 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:41.182657 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:41.182668 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:41.182676 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:41.183047 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:41.183351 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:41.183356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:41.183359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:41.183482 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:41.183946 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:41.183953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:41.184137 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.184275 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.184541 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":271,"threads_started":1}
I20260924 10:01:41.189752 15701 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:37282:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:01:41.190730 15701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:41.196987 15977 tablet_service.cc:1467] Processing CreateTablet for tablet 753e803d09e34914a953cc653360af54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8bfe290f8a4e472380aec8eba084484a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:01:41.196867 15838 tablet_service.cc:1467] Processing CreateTablet for tablet 753e803d09e34914a953cc653360af54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8bfe290f8a4e472380aec8eba084484a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:01:41.197160 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 753e803d09e34914a953cc653360af54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8bfe290f8a4e472380aec8eba084484a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:01:41.197777 15977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753e803d09e34914a953cc653360af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.197824 15838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753e803d09e34914a953cc653360af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.197821 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753e803d09e34914a953cc653360af54. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 10:01:41.198477 16048 tablet.cc:2369] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:01:41.203768 16205 tablet_bootstrap.cc:492] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Bootstrap starting.
I20260924 10:01:41.204538 16206 tablet_bootstrap.cc:492] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: Bootstrap starting.
I20260924 10:01:41.204738 16207 tablet_bootstrap.cc:492] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: Bootstrap starting.
I20260924 10:01:41.205492 16205 tablet_bootstrap.cc:654] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.206048 16206 tablet_bootstrap.cc:654] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.206125 16207 tablet_bootstrap.cc:654] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.208190 16207 tablet_bootstrap.cc:492] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: No bootstrap required, opened a new log
I20260924 10:01:41.208287 16206 tablet_bootstrap.cc:492] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: No bootstrap required, opened a new log
I20260924 10:01:41.208335 16207 ts_tablet_manager.cc:1397] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 10:01:41.208408 16206 ts_tablet_manager.cc:1397] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 10:01:41.208523 16205 tablet_bootstrap.cc:492] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: No bootstrap required, opened a new log
I20260924 10:01:41.208667 16205 ts_tablet_manager.cc:1397] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 10:01:41.208966 16207 raft_consensus.cc:348] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.209133 16207 raft_consensus.cc:374] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.209213 16207 raft_consensus.cc:729] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.209107 16206 raft_consensus.cc:348] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.209327 16206 raft_consensus.cc:374] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.209414 16206 raft_consensus.cc:729] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75154b9842f147a3bb0018ba890cc0be, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.209343 16205 raft_consensus.cc:348] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.209393 16207 consensus_queue.cc:260] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.209556 16205 raft_consensus.cc:374] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.209633 16205 raft_consensus.cc:729] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.209612 16206 consensus_queue.cc:260] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.209807 16211 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.209991 16207 ts_tablet_manager.cc:1428] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:01:41.209882 16205 consensus_queue.cc:260] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.210074 16211 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.210144 16206 ts_tablet_manager.cc:1428] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:01:41.210263 16212 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.210619 16212 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.210427 16213 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.210729 16213 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.211081 16205 ts_tablet_manager.cc:1428] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 10:01:41.217204 16212 raft_consensus.cc:482] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:41.217383 16212 raft_consensus.cc:504] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.218827 16212 leader_election.cc:288] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:41.226711 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.219915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:37221 (local address 127.13.162.3:45049)
0924 10:01:41.220099 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.220104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.220379 (+ 275us) server_negotiation.cc:407] Connection header received
0924 10:01:41.220501 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.220506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.220558 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.220625 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:41.221359 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.222267 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.224489 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.225643 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.225769 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.225828 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.226301 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.226453 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:01:41.226732 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.219203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:38301 (local address 127.13.162.1:33959)
0924 10:01:41.219356 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.219363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.219822 (+ 459us) server_negotiation.cc:407] Connection header received
0924 10:01:41.220030 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.220036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.220123 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.220185 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:01:41.220923 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.221960 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.224408 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.225581 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.225748 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.225819 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.226297 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.226485 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:01:41.226944 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.219753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:45049 (local address 127.13.162.2:37221)
0924 10:01:41.220292 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.220309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.220409 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.220657 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.220665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.220687 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:41.221208 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.221219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.222420 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.222426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.224365 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.224375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.225869 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.225875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.226132 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.226177 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.226263 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.226498 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":178,"thread_start_us":267,"threads_started":1}
I20260924 10:01:41.226967 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.219127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33959 (local address 127.13.162.2:38301)
0924 10:01:41.219659 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.219676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.219854 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.220385 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.220389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.220464 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:01:41.220778 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.220794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.222084 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.222088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.224259 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.224277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.225845 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.225856 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.226127 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.226183 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.226273 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.226493 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":83,"spinlock_wait_cycles":23808,"thread_start_us":302,"threads_started":1}
I20260924 10:01:41.227705 15863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e803d09e34914a953cc653360af54" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be" is_pre_election: true
I20260924 10:01:41.227973 15863 raft_consensus.cc:2454] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 0.
I20260924 10:01:41.228008 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e803d09e34914a953cc653360af54" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b97c26408b748c7a730bad7246ed699" is_pre_election: true
I20260924 10:01:41.228407 16136 raft_consensus.cc:2454] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 0.
I20260924 10:01:41.228468 15930 leader_election.cc:302] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:41.228889 16212 raft_consensus.cc:2781] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:41.229014 16212 raft_consensus.cc:482] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:41.229112 16212 raft_consensus.cc:3037] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.231165 16212 raft_consensus.cc:504] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.231660 16212 leader_election.cc:288] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 election: Requested vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:41.232079 15863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e803d09e34914a953cc653360af54" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be"
I20260924 10:01:41.232096 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e803d09e34914a953cc653360af54" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b97c26408b748c7a730bad7246ed699"
I20260924 10:01:41.232235 16136 raft_consensus.cc:3037] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.232288 15863 raft_consensus.cc:3037] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Advancing to term 1
W20260924 10:01:41.233100 15909 tablet.cc:2369] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:01:41.234261 16136 raft_consensus.cc:2454] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 1.
I20260924 10:01:41.234618 15930 leader_election.cc:302] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b97c26408b748c7a730bad7246ed699, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:41.234763 15863 raft_consensus.cc:2454] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 1.
I20260924 10:01:41.234839 16212 raft_consensus.cc:2781] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:41.235293 16212 raft_consensus.cc:686] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 LEADER]: Becoming Leader. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Running, Role: LEADER
I20260924 10:01:41.235716 16212 consensus_queue.cc:237] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.238701 16228 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: TxnStatusTablet state changed. Reason: New leader 8ce5e1de5fd043a5b6b99a4f49a37d7d. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.239037 16228 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:01:41.239698 16230 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:01:41.239946 16230 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:01:41.240408 15699 catalog_manager.cc:5495] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2). New cstate: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260924 10:01:41.261969 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.256014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42345 (local address 127.0.0.1:42754)
0924 10:01:41.256225 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.256245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.256364 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.256849 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.256852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.256862 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:41.257006 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.257013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.258369 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.258373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.259422 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.259434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.260384 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.260389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.260535 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.260588 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.261556 (+ 968us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.261573 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.261639 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.261750 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:01:41.262053 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.256033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.13.162.2:42345)
0924 10:01:41.256204 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.256210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.256311 (+ 101us) server_negotiation.cc:407] Connection header received
0924 10:01:41.256533 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.256539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.256665 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.256835 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:41.257115 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.258231 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.259548 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.260228 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.260355 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.260763 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.261449 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.261585 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.261663 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.261742 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:01:41.270347 16136 raft_consensus.cc:1261] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Refusing update from remote peer 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:41.270467 15863 raft_consensus.cc:1261] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Refusing update from remote peer 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:41.271271 16212 consensus_queue.cc:1035] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:41.271589 16228 consensus_queue.cc:1035] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:41.287009 16213 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: TxnStatusTablet state changed. Reason: New leader 8ce5e1de5fd043a5b6b99a4f49a37d7d. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.287286 16213 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.290571 16233 mvcc.cc:205] Tried to move back new op lower bound from 7332839838794563584 to 7332839838664953856. Current Snapshot: MvccSnapshot[applied={T|T < 7332839838794563584}]
I20260924 10:01:41.291116 16234 mvcc.cc:205] Tried to move back new op lower bound from 7332839838794563584 to 7332839838664953856. Current Snapshot: MvccSnapshot[applied={T|T < 7332839838794563584}]
I20260924 10:01:41.299588 16211 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: TxnStatusTablet state changed. Reason: New leader 8ce5e1de5fd043a5b6b99a4f49a37d7d. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.299896 16211 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.301167 16212 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.301430 16212 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:01:41.303820 16236 mvcc.cc:205] Tried to move back new op lower bound from 7332839838794563584 to 7332839838664953856. Current Snapshot: MvccSnapshot[applied={T|T < 7332839838794563584}]
I20260924 10:01:41.304430 16212 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.304662 16212 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:01:41.304776 16213 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.304989 16213 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 75154b9842f147a3bb0018ba890cc0be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.308512 16211 tablet_replica.cc:439] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } } }
I20260924 10:01:41.308794 16211 tablet_replica.cc:442] T 753e803d09e34914a953cc653360af54 P 7b97c26408b748c7a730bad7246ed699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:01:41.319124 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.313877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.13.162.3:45049)
0924 10:01:41.314017 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.314023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.314223 (+ 200us) server_negotiation.cc:407] Connection header received
0924 10:01:41.314462 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.314467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.314540 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.314661 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:41.315057 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.315770 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.317074 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.317466 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.317569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.317975 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.318527 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.318668 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.318755 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.318831 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:01:41.334673 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.329324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:51923 (local address 127.13.162.2:42345)
0924 10:01:41.329529 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.329535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.329595 (+ 60us) server_negotiation.cc:407] Connection header received
0924 10:01:41.329774 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.329778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.329848 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.329942 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:41.330292 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.331153 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.333003 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.333466 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.333565 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.334075 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.334181 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.334251 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.334394 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.334456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:01:41.334831 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.329312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42345 (local address 127.13.162.3:51923)
0924 10:01:41.329524 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.329543 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.329670 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.329952 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.329957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.329970 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:41.330143 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.330153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.331292 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.331298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.332853 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.332876 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.333606 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.333612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.333784 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.333852 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.334268 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.334278 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.334377 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.334523 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:01:41.344190 15862 raft_consensus.cc:1261] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Refusing update from remote peer 7b97c26408b748c7a730bad7246ed699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:41.344715 15997 raft_consensus.cc:1261] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Refusing update from remote peer 7b97c26408b748c7a730bad7246ed699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:41.345443 16211 consensus_queue.cc:1035] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:41.345721 16215 consensus_queue.cc:1035] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:41.384063 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.377495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:41519 (local address 127.13.162.3:45049)
0924 10:01:41.377676 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.377682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.378265 (+ 583us) server_negotiation.cc:407] Connection header received
0924 10:01:41.378573 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.378579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.378661 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.378720 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:41.379496 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.380380 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.381990 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.382586 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.382672 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.383264 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.383409 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.383465 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.383685 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.383756 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:01:41.384410 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.377411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:45049 (local address 127.13.162.2:41519)
0924 10:01:41.378176 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.378195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.378418 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.378918 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.378922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.378933 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:41.379331 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.379344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.380520 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.380524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.381854 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.381868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.382855 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.382861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.383018 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.383086 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.383543 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.383552 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.383660 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.383790 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":2494,"threads_started":1}
I20260924 10:01:41.441876 15700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:41.442170 15700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:41.447423 15700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Sending DeleteTablet for 3 replicas of tablet 90c2ae829ef54f67b71c24686c26af0d
I20260924 10:01:41.448500 15977 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c2ae829ef54f67b71c24686c26af0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:41 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:42738
I20260924 10:01:41.449129 15838 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c2ae829ef54f67b71c24686c26af0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:41 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:53924
I20260924 10:01:41.450893 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c2ae829ef54f67b71c24686c26af0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:41 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260924 10:01:41.452051 15700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:41.453191 15700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:41.462291 15976 tablet_service.cc:1467] Processing CreateTablet for tablet 55e9063790ec4b47a7609d8bec5592d6 (DEFAULT_TABLE table=test-table [id=c76347fb3bc245dcac079b94a2aff28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:41.462746 15837 tablet_service.cc:1467] Processing CreateTablet for tablet 55e9063790ec4b47a7609d8bec5592d6 (DEFAULT_TABLE table=test-table [id=c76347fb3bc245dcac079b94a2aff28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:41.462918 15976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e9063790ec4b47a7609d8bec5592d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.462975 16115 tablet_service.cc:1467] Processing CreateTablet for tablet 55e9063790ec4b47a7609d8bec5592d6 (DEFAULT_TABLE table=test-table [id=c76347fb3bc245dcac079b94a2aff28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:41.463342 15837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e9063790ec4b47a7609d8bec5592d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.463397 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e9063790ec4b47a7609d8bec5592d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:41.466408 16253 tablet_replica.cc:330] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: stopping tablet replica
I20260924 10:01:41.466883 16253 raft_consensus.cc:2229] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:41.467272 16253 raft_consensus.cc:2258] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:41.469331 16205 tablet_bootstrap.cc:492] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Bootstrap starting.
I20260924 10:01:41.469877 16206 tablet_bootstrap.cc:492] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Bootstrap starting.
I20260924 10:01:41.471031 16205 tablet_bootstrap.cc:654] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.471392 16206 tablet_bootstrap.cc:654] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.471618 16207 tablet_bootstrap.cc:492] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Bootstrap starting.
I20260924 10:01:41.473179 16207 tablet_bootstrap.cc:654] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:41.475997 16253 ts_tablet_manager.cc:1905] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:41.483870 16253 ts_tablet_manager.cc:1918] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:41.484160 16206 tablet_bootstrap.cc:492] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: No bootstrap required, opened a new log
I20260924 10:01:41.491935 16206 ts_tablet_manager.cc:1397] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20260924 10:01:41.492866 16206 raft_consensus.cc:348] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.493229 16206 raft_consensus.cc:374] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.493361 16206 raft_consensus.cc:729] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75154b9842f147a3bb0018ba890cc0be, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.494057 16206 consensus_queue.cc:260] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.497042 16206 ts_tablet_manager.cc:1428] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 10:01:41.491778 16253 log.cc:1196] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/90c2ae829ef54f67b71c24686c26af0d
I20260924 10:01:41.499696 16207 tablet_bootstrap.cc:492] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: No bootstrap required, opened a new log
I20260924 10:01:41.499905 16253 ts_tablet_manager.cc:1939] T 90c2ae829ef54f67b71c24686c26af0d P 75154b9842f147a3bb0018ba890cc0be: Deleting consensus metadata
I20260924 10:01:41.499966 16207 ts_tablet_manager.cc:1397] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.001s
I20260924 10:01:41.500852 16207 raft_consensus.cc:348] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.501173 16207 raft_consensus.cc:374] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.501297 16207 raft_consensus.cc:729] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.501624 16207 consensus_queue.cc:260] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.502307 15687 catalog_manager.cc:4841] TS 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959): tablet 90c2ae829ef54f67b71c24686c26af0d (table test-table [id=c007e8cdaa314070811f019fa95afdbe]) successfully deleted
I20260924 10:01:41.504549 16207 ts_tablet_manager.cc:1428] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:01:41.508009 16254 tablet_replica.cc:330] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: stopping tablet replica
I20260924 10:01:41.508453 16254 raft_consensus.cc:2229] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:01:41.509791 16254 raft_consensus.cc:2258] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:41.511734 16252 tablet_replica.cc:330] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: stopping tablet replica
I20260924 10:01:41.511902 16254 ts_tablet_manager.cc:1905] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:41.512203 16252 raft_consensus.cc:2229] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:41.512588 16252 raft_consensus.cc:2258] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:41.519309 16205 tablet_bootstrap.cc:492] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: No bootstrap required, opened a new log
I20260924 10:01:41.519380 16252 ts_tablet_manager.cc:1905] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:41.519488 16205 ts_tablet_manager.cc:1397] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.016s
I20260924 10:01:41.520402 16205 raft_consensus.cc:348] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.520712 16205 raft_consensus.cc:374] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:41.520900 16205 raft_consensus.cc:729] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Initialized, Role: FOLLOWER
I20260924 10:01:41.521234 16205 consensus_queue.cc:260] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.524183 16205 ts_tablet_manager.cc:1428] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 10:01:41.528631 16254 ts_tablet_manager.cc:1918] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:41.529508 16254 log.cc:1196] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/90c2ae829ef54f67b71c24686c26af0d
I20260924 10:01:41.530176 16254 ts_tablet_manager.cc:1939] T 90c2ae829ef54f67b71c24686c26af0d P 7b97c26408b748c7a730bad7246ed699: Deleting consensus metadata
I20260924 10:01:41.532061 15687 catalog_manager.cc:4841] TS 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049): tablet 90c2ae829ef54f67b71c24686c26af0d (table test-table [id=c007e8cdaa314070811f019fa95afdbe]) successfully deleted
I20260924 10:01:41.534670 16252 ts_tablet_manager.cc:1918] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:41.534907 16252 log.cc:1196] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/90c2ae829ef54f67b71c24686c26af0d
I20260924 10:01:41.535478 16252 ts_tablet_manager.cc:1939] T 90c2ae829ef54f67b71c24686c26af0d P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting consensus metadata
I20260924 10:01:41.537158 15688 catalog_manager.cc:4841] TS 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345): tablet 90c2ae829ef54f67b71c24686c26af0d (table test-table [id=c007e8cdaa314070811f019fa95afdbe]) successfully deleted
I20260924 10:01:41.832224 16244 raft_consensus.cc:482] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:41.832451 16244 raft_consensus.cc:504] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.833233 16244 leader_election.cc:288] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:41.833799 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9063790ec4b47a7609d8bec5592d6" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be" is_pre_election: true
I20260924 10:01:41.833863 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9063790ec4b47a7609d8bec5592d6" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b97c26408b748c7a730bad7246ed699" is_pre_election: true
I20260924 10:01:41.834076 15862 raft_consensus.cc:2454] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 0.
I20260924 10:01:41.834122 16136 raft_consensus.cc:2454] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 0.
I20260924 10:01:41.834545 15930 leader_election.cc:302] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:41.834825 16244 raft_consensus.cc:2781] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:41.834951 16244 raft_consensus.cc:482] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:41.835047 16244 raft_consensus.cc:3037] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.836961 16244 raft_consensus.cc:504] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.837476 16244 leader_election.cc:288] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 election: Requested vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:41.837939 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9063790ec4b47a7609d8bec5592d6" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be"
I20260924 10:01:41.838009 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e9063790ec4b47a7609d8bec5592d6" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b97c26408b748c7a730bad7246ed699"
I20260924 10:01:41.838124 15862 raft_consensus.cc:3037] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.838236 16136 raft_consensus.cc:3037] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:41.840037 15862 raft_consensus.cc:2454] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 1.
I20260924 10:01:41.840381 15930 leader_election.cc:302] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:41.840612 16244 raft_consensus.cc:2781] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:41.840771 16244 raft_consensus.cc:686] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 LEADER]: Becoming Leader. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Running, Role: LEADER
I20260924 10:01:41.841003 16244 consensus_queue.cc:237] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:41.841548 16136 raft_consensus.cc:2454] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 1.
I20260924 10:01:41.851212 15700 catalog_manager.cc:5495] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2). New cstate: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260924 10:01:41.957748 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.953050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.13.162.2:42345)
0924 10:01:41.953721 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.953726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.953745 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:01:41.953906 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.953910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.953970 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.954047 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:41.954518 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.955179 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.956395 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.956783 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.956874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.957161 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.957250 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.957309 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.957419 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.957475 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":438,"threads_started":1}
I20260924 10:01:41.969432 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.964928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:39379 (local address 127.13.162.2:42345)
0924 10:01:41.965074 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:41.965080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:41.965490 (+ 410us) server_negotiation.cc:407] Connection header received
0924 10:01:41.965630 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:41.965635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:41.965688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:41.965753 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:41.966098 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.966836 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.968012 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.968386 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.968482 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.968795 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.968880 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.968948 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:41.969058 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 10:01:41.969117 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:01:41.969457 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:41.964920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42345 (local address 127.13.162.2:39379)
0924 10:01:41.965402 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:41.965421 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:41.965532 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:41.965773 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:41.965777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:41.965786 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:01:41.965982 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.965989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.966950 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.966955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.967894 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:41.967904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:41.968518 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:41.968524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:41.968636 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:41.968665 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:01:41.968960 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:01:41.968970 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:01:41.969038 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:01:41.969153 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260924 10:01:41.975955 16136 raft_consensus.cc:1261] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Refusing update from remote peer 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:41.975988 15862 raft_consensus.cc:1261] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Refusing update from remote peer 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:41.976441 16262 consensus_queue.cc:1035] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:41.976681 16244 consensus_queue.cc:1035] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:01:42.000579 16259 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:01:43.611222 15699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:43.611503 15699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:43.620677 15699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Sending DeleteTablet for 3 replicas of tablet 55e9063790ec4b47a7609d8bec5592d6
I20260924 10:01:43.621619 15976 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9063790ec4b47a7609d8bec5592d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:42738
I20260924 10:01:43.621935 15837 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9063790ec4b47a7609d8bec5592d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:53924
I20260924 10:01:43.622272 16295 tablet_replica.cc:330] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: stopping tablet replica
I20260924 10:01:43.622540 16295 raft_consensus.cc:2229] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:01:43.622671 16296 tablet_replica.cc:330] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: stopping tablet replica
I20260924 10:01:43.622849 16296 raft_consensus.cc:2229] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:43.623065 16296 raft_consensus.cc:2258] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:43.623060 16295 raft_consensus.cc:2258] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:43.623904 16295 ts_tablet_manager.cc:1905] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:43.624234 16296 ts_tablet_manager.cc:1905] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:43.626504 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e9063790ec4b47a7609d8bec5592d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260924 10:01:43.627110 16297 tablet_replica.cc:330] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: stopping tablet replica
I20260924 10:01:43.627351 16297 raft_consensus.cc:2229] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:43.627624 16297 raft_consensus.cc:2258] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:43.628758 16297 ts_tablet_manager.cc:1905] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:43.629247 16295 ts_tablet_manager.cc:1918] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:01:43.629436 16295 log.cc:1196] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/55e9063790ec4b47a7609d8bec5592d6
I20260924 10:01:43.629930 16295 ts_tablet_manager.cc:1939] T 55e9063790ec4b47a7609d8bec5592d6 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting consensus metadata
I20260924 10:01:43.630307 16296 ts_tablet_manager.cc:1918] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:01:43.630455 16296 log.cc:1196] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/55e9063790ec4b47a7609d8bec5592d6
I20260924 10:01:43.630885 16296 ts_tablet_manager.cc:1939] T 55e9063790ec4b47a7609d8bec5592d6 P 75154b9842f147a3bb0018ba890cc0be: Deleting consensus metadata
I20260924 10:01:43.631394 15688 catalog_manager.cc:4841] TS 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345): tablet 55e9063790ec4b47a7609d8bec5592d6 (table test-table [id=c76347fb3bc245dcac079b94a2aff28e]) successfully deleted
I20260924 10:01:43.631963 15687 catalog_manager.cc:4841] TS 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959): tablet 55e9063790ec4b47a7609d8bec5592d6 (table test-table [id=c76347fb3bc245dcac079b94a2aff28e]) successfully deleted
I20260924 10:01:43.634895 16297 ts_tablet_manager.cc:1918] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:01:43.635012 16297 log.cc:1196] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/55e9063790ec4b47a7609d8bec5592d6
I20260924 10:01:43.635385 16297 ts_tablet_manager.cc:1939] T 55e9063790ec4b47a7609d8bec5592d6 P 7b97c26408b748c7a730bad7246ed699: Deleting consensus metadata
I20260924 10:01:43.636293 15687 catalog_manager.cc:4841] TS 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049): tablet 55e9063790ec4b47a7609d8bec5592d6 (table test-table [id=c76347fb3bc245dcac079b94a2aff28e]) successfully deleted
Sep 24 10:01:43 dist-test-slave-jvsj krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:01:43 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:43.670495 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:43.657352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.13.162.62:39197)
0924 10:01:43.657922 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:43.657926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:43.657948 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:01:43.658063 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:43.658067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:43.658125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:43.658210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:43.659551 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:43.660420 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:43.661694 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:43.662111 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:43.662203 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:43.665798 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:43.665803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:43.665807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:43.665841 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:43.668252 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:43.668858 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:43.668861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:43.668864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:43.668980 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:43.669257 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:43.669260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:43.669261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:43.669525 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:43.669697 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:43.669987 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 10:01:43.670144 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":296,"threads_started":1}
I20260924 10:01:43.673815 15699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:43.674459 15699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:43.680768 15976 tablet_service.cc:1467] Processing CreateTablet for tablet 05869a94adad448f92ea1880223bcdf2 (DEFAULT_TABLE table=test-table [id=ccf9749c65954bb898913fec075b6263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:43.680771 15837 tablet_service.cc:1467] Processing CreateTablet for tablet 05869a94adad448f92ea1880223bcdf2 (DEFAULT_TABLE table=test-table [id=ccf9749c65954bb898913fec075b6263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:43.680848 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 05869a94adad448f92ea1880223bcdf2 (DEFAULT_TABLE table=test-table [id=ccf9749c65954bb898913fec075b6263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:43.681190 15976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05869a94adad448f92ea1880223bcdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:43.681205 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05869a94adad448f92ea1880223bcdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:43.681222 15837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05869a94adad448f92ea1880223bcdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:43.686201 16306 tablet_bootstrap.cc:492] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Bootstrap starting.
I20260924 10:01:43.687811 16306 tablet_bootstrap.cc:654] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:43.689793 16306 tablet_bootstrap.cc:492] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: No bootstrap required, opened a new log
I20260924 10:01:43.689922 16306 ts_tablet_manager.cc:1397] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 10:01:43.690721 16306 raft_consensus.cc:348] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.691035 16306 raft_consensus.cc:374] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:43.691174 16306 raft_consensus.cc:729] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Initialized, Role: FOLLOWER
I20260924 10:01:43.691401 16306 consensus_queue.cc:260] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.691902 16306 ts_tablet_manager.cc:1428] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:01:43.695879 16308 tablet_bootstrap.cc:492] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Bootstrap starting.
I20260924 10:01:43.696280 16307 tablet_bootstrap.cc:492] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Bootstrap starting.
I20260924 10:01:43.697600 16308 tablet_bootstrap.cc:654] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:43.697894 16307 tablet_bootstrap.cc:654] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:43.700063 16308 tablet_bootstrap.cc:492] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: No bootstrap required, opened a new log
I20260924 10:01:43.700210 16308 ts_tablet_manager.cc:1397] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 10:01:43.700580 16307 tablet_bootstrap.cc:492] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: No bootstrap required, opened a new log
I20260924 10:01:43.700724 16307 ts_tablet_manager.cc:1397] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:01:43.700995 16308 raft_consensus.cc:348] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.701257 16308 raft_consensus.cc:374] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:43.701364 16308 raft_consensus.cc:729] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Initialized, Role: FOLLOWER
I20260924 10:01:43.701452 16307 raft_consensus.cc:348] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.701666 16307 raft_consensus.cc:374] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:43.701597 16308 consensus_queue.cc:260] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.701752 16307 raft_consensus.cc:729] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75154b9842f147a3bb0018ba890cc0be, State: Initialized, Role: FOLLOWER
I20260924 10:01:43.702000 16307 consensus_queue.cc:260] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.702776 16307 ts_tablet_manager.cc:1428] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 10:01:43.702937 16308 ts_tablet_manager.cc:1428] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:43.729622 16283 raft_consensus.cc:482] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:43.729825 16283 raft_consensus.cc:504] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.730510 16283 leader_election.cc:288] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:43.730971 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05869a94adad448f92ea1880223bcdf2" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be" is_pre_election: true
I20260924 10:01:43.731057 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05869a94adad448f92ea1880223bcdf2" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b97c26408b748c7a730bad7246ed699" is_pre_election: true
I20260924 10:01:43.731227 15862 raft_consensus.cc:2454] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 0.
I20260924 10:01:43.731299 16136 raft_consensus.cc:2454] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 0.
I20260924 10:01:43.731709 15930 leader_election.cc:302] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:43.731994 16283 raft_consensus.cc:2781] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:43.732110 16283 raft_consensus.cc:482] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:43.732195 16283 raft_consensus.cc:3037] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:43.734220 16283 raft_consensus.cc:504] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.734792 16283 leader_election.cc:288] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 election: Requested vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049)
I20260924 10:01:43.735234 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05869a94adad448f92ea1880223bcdf2" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be"
I20260924 10:01:43.735308 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05869a94adad448f92ea1880223bcdf2" candidate_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b97c26408b748c7a730bad7246ed699"
I20260924 10:01:43.735430 15862 raft_consensus.cc:3037] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:43.735495 16136 raft_consensus.cc:3037] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:43.737768 15862 raft_consensus.cc:2454] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 1.
I20260924 10:01:43.737862 16136 raft_consensus.cc:2454] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5e1de5fd043a5b6b99a4f49a37d7d in term 1.
I20260924 10:01:43.738155 15930 leader_election.cc:302] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 8ce5e1de5fd043a5b6b99a4f49a37d7d; no voters:
I20260924 10:01:43.738458 16283 raft_consensus.cc:2781] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:43.738595 16283 raft_consensus.cc:686] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 LEADER]: Becoming Leader. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Running, Role: LEADER
I20260924 10:01:43.738847 16283 consensus_queue.cc:237] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.741287 15699 catalog_manager.cc:5495] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2). New cstate: current_term: 1 leader_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260924 10:01:43.766706 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:43.762067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.13.162.2:42345)
0924 10:01:43.762626 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:43.762631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:43.762649 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:01:43.762696 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:43.762699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:43.762766 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:43.762841 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:43.763175 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:43.763961 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:43.765147 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:43.765549 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:43.765637 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:43.766012 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:43.766129 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:43.766201 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:43.766302 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:01:43.766399 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":317,"threads_started":1}
I20260924 10:01:43.774746 16136 raft_consensus.cc:1261] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Refusing update from remote peer 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:43.774853 15862 raft_consensus.cc:1261] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Refusing update from remote peer 8ce5e1de5fd043a5b6b99a4f49a37d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:43.775220 16283 consensus_queue.cc:1035] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:43.775442 16262 consensus_queue.cc:1035] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:43.814383 15700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:43.814595 15700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:43.817385 15700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Sending DeleteTablet for 3 replicas of tablet 05869a94adad448f92ea1880223bcdf2
I20260924 10:01:43.818120 15837 tablet_service.cc:1514] Processing DeleteTablet for tablet 05869a94adad448f92ea1880223bcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:53924
I20260924 10:01:43.818253 15977 tablet_service.cc:1514] Processing DeleteTablet for tablet 05869a94adad448f92ea1880223bcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:42738
I20260924 10:01:43.818351 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 05869a94adad448f92ea1880223bcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260924 10:01:43.818456 16296 tablet_replica.cc:330] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: stopping tablet replica
I20260924 10:01:43.818548 16295 tablet_replica.cc:330] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: stopping tablet replica
I20260924 10:01:43.818639 16297 tablet_replica.cc:330] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: stopping tablet replica
I20260924 10:01:43.818643 16296 raft_consensus.cc:2229] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:43.818712 16295 raft_consensus.cc:2229] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:01:43.818816 16296 raft_consensus.cc:2258] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:43.818822 16297 raft_consensus.cc:2229] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:43.819002 16297 raft_consensus.cc:2258] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:43.819012 16295 raft_consensus.cc:2258] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:43.819856 16296 ts_tablet_manager.cc:1905] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:43.819924 16297 ts_tablet_manager.cc:1905] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:43.819770 15700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:01:43.820406 15700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:01:43.821043 16295 ts_tablet_manager.cc:1905] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:43.827062 16295 ts_tablet_manager.cc:1918] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:43.827009 15837 tablet_service.cc:1467] Processing CreateTablet for tablet 977fd50bbff045de8adbb9d0a101ab36 (DEFAULT_TABLE table=test-table [id=3a1044e6dae7443385d03df1de814ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:43.827297 16295 log.cc:1196] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/05869a94adad448f92ea1880223bcdf2
I20260924 10:01:43.827420 15837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977fd50bbff045de8adbb9d0a101ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:43.827785 16295 ts_tablet_manager.cc:1939] T 05869a94adad448f92ea1880223bcdf2 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting consensus metadata
I20260924 10:01:43.828120 16297 ts_tablet_manager.cc:1918] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:43.828269 16297 log.cc:1196] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/05869a94adad448f92ea1880223bcdf2
I20260924 10:01:43.828619 16297 ts_tablet_manager.cc:1939] T 05869a94adad448f92ea1880223bcdf2 P 7b97c26408b748c7a730bad7246ed699: Deleting consensus metadata
I20260924 10:01:43.828974 15688 catalog_manager.cc:4841] TS 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345): tablet 05869a94adad448f92ea1880223bcdf2 (table test-table [id=ccf9749c65954bb898913fec075b6263]) successfully deleted
I20260924 10:01:43.829704 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 977fd50bbff045de8adbb9d0a101ab36 (DEFAULT_TABLE table=test-table [id=3a1044e6dae7443385d03df1de814ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:43.830094 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977fd50bbff045de8adbb9d0a101ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:43.831025 16296 ts_tablet_manager.cc:1918] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:01:43.831159 16296 log.cc:1196] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/05869a94adad448f92ea1880223bcdf2
I20260924 10:01:43.831620 16296 ts_tablet_manager.cc:1939] T 05869a94adad448f92ea1880223bcdf2 P 75154b9842f147a3bb0018ba890cc0be: Deleting consensus metadata
I20260924 10:01:43.831698 15977 tablet_service.cc:1467] Processing CreateTablet for tablet 977fd50bbff045de8adbb9d0a101ab36 (DEFAULT_TABLE table=test-table [id=3a1044e6dae7443385d03df1de814ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:01:43.831991 15977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977fd50bbff045de8adbb9d0a101ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:43.832535 15687 catalog_manager.cc:4841] TS 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959): tablet 05869a94adad448f92ea1880223bcdf2 (table test-table [id=ccf9749c65954bb898913fec075b6263]) successfully deleted
I20260924 10:01:43.833642 15687 catalog_manager.cc:4841] TS 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049): tablet 05869a94adad448f92ea1880223bcdf2 (table test-table [id=ccf9749c65954bb898913fec075b6263]) successfully deleted
I20260924 10:01:43.835106 16307 tablet_bootstrap.cc:492] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Bootstrap starting.
I20260924 10:01:43.835772 16308 tablet_bootstrap.cc:492] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Bootstrap starting.
I20260924 10:01:43.836704 16307 tablet_bootstrap.cc:654] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:43.836948 16308 tablet_bootstrap.cc:654] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:43.837208 16306 tablet_bootstrap.cc:492] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Bootstrap starting.
I20260924 10:01:43.838790 16306 tablet_bootstrap.cc:654] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:43.839789 16307 tablet_bootstrap.cc:492] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: No bootstrap required, opened a new log
I20260924 10:01:43.839941 16307 ts_tablet_manager.cc:1397] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 10:01:43.840268 16308 tablet_bootstrap.cc:492] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: No bootstrap required, opened a new log
I20260924 10:01:43.840404 16308 ts_tablet_manager.cc:1397] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:01:43.840601 16306 tablet_bootstrap.cc:492] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: No bootstrap required, opened a new log
I20260924 10:01:43.840770 16306 ts_tablet_manager.cc:1397] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 10:01:43.840679 16307 raft_consensus.cc:348] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.840932 16307 raft_consensus.cc:374] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:43.841010 16307 raft_consensus.cc:729] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75154b9842f147a3bb0018ba890cc0be, State: Initialized, Role: FOLLOWER
I20260924 10:01:43.841007 16308 raft_consensus.cc:348] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.841245 16308 raft_consensus.cc:374] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:43.841320 16308 raft_consensus.cc:729] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Initialized, Role: FOLLOWER
I20260924 10:01:43.841228 16307 consensus_queue.cc:260] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.841451 16306 raft_consensus.cc:348] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.841559 16308 consensus_queue.cc:260] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.841749 16307 ts_tablet_manager.cc:1428] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 10:01:43.842172 16308 ts_tablet_manager.cc:1428] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:01:43.842803 16306 raft_consensus.cc:374] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:43.842904 16306 raft_consensus.cc:729] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5e1de5fd043a5b6b99a4f49a37d7d, State: Initialized, Role: FOLLOWER
I20260924 10:01:43.843124 16306 consensus_queue.cc:260] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:43.843743 16306 ts_tablet_manager.cc:1428] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:44.090579 16312 raft_consensus.cc:482] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:01:44.090819 16312 raft_consensus.cc:504] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:44.091444 16312 leader_election.cc:288] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345)
I20260924 10:01:44.091985 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977fd50bbff045de8adbb9d0a101ab36" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be" is_pre_election: true
I20260924 10:01:44.092007 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977fd50bbff045de8adbb9d0a101ab36" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" is_pre_election: true
I20260924 10:01:44.092216 15862 raft_consensus.cc:2454] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 0.
I20260924 10:01:44.092221 15996 raft_consensus.cc:2454] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 0.
I20260924 10:01:44.092646 16069 leader_election.cc:302] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 7b97c26408b748c7a730bad7246ed699; no voters:
I20260924 10:01:44.092891 16312 raft_consensus.cc:2781] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:01:44.092988 16312 raft_consensus.cc:482] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:01:44.093077 16312 raft_consensus.cc:3037] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:44.095100 16312 raft_consensus.cc:504] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:44.095642 16312 leader_election.cc:288] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 election: Requested vote from peers 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959), 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345)
I20260924 10:01:44.096125 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977fd50bbff045de8adbb9d0a101ab36" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d"
I20260924 10:01:44.096108 15862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977fd50bbff045de8adbb9d0a101ab36" candidate_uuid: "7b97c26408b748c7a730bad7246ed699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75154b9842f147a3bb0018ba890cc0be"
I20260924 10:01:44.096308 15996 raft_consensus.cc:3037] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:44.096359 15862 raft_consensus.cc:3037] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:44.099139 15862 raft_consensus.cc:2454] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 1.
I20260924 10:01:44.099154 15996 raft_consensus.cc:2454] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b97c26408b748c7a730bad7246ed699 in term 1.
I20260924 10:01:44.099509 16069 leader_election.cc:302] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75154b9842f147a3bb0018ba890cc0be, 7b97c26408b748c7a730bad7246ed699; no voters:
I20260924 10:01:44.099788 16312 raft_consensus.cc:2781] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:44.100180 16312 raft_consensus.cc:686] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 1 LEADER]: Becoming Leader. State: Replica: 7b97c26408b748c7a730bad7246ed699, State: Running, Role: LEADER
I20260924 10:01:44.100435 16312 consensus_queue.cc:237] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } }
I20260924 10:01:44.102805 15701 catalog_manager.cc:5495] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b97c26408b748c7a730bad7246ed699 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "7b97c26408b748c7a730bad7246ed699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b97c26408b748c7a730bad7246ed699" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45049 } health_report { overall_health: HEALTHY } } }
I20260924 10:01:44.136135 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:44.131365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.13.162.3:45049)
0924 10:01:44.131940 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:44.131944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:44.131960 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:01:44.132005 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:44.132008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:44.132062 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:44.132140 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:44.132542 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:44.133276 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:44.134542 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:44.134940 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:44.135043 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:44.135444 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:44.135544 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:44.135614 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:44.135745 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:01:44.135817 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":321,"threads_started":1}
I20260924 10:01:44.144069 15996 raft_consensus.cc:1261] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Refusing update from remote peer 7b97c26408b748c7a730bad7246ed699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:44.144080 15862 raft_consensus.cc:1261] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Refusing update from remote peer 7b97c26408b748c7a730bad7246ed699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:44.144568 16323 consensus_queue.cc:1035] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75154b9842f147a3bb0018ba890cc0be" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:44.144850 16312 consensus_queue.cc:1035] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce5e1de5fd043a5b6b99a4f49a37d7d" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:01:44.161623 16279 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:01:45.735368 15699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:45.735608 15699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:01:45.739566 15699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62eb140e4790414ea03311ce38309820: Sending DeleteTablet for 3 replicas of tablet 977fd50bbff045de8adbb9d0a101ab36
I20260924 10:01:45.740326 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 977fd50bbff045de8adbb9d0a101ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:45 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260924 10:01:45.740319 15837 tablet_service.cc:1514] Processing DeleteTablet for tablet 977fd50bbff045de8adbb9d0a101ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:45 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:53924
I20260924 10:01:45.740370 15977 tablet_service.cc:1514] Processing DeleteTablet for tablet 977fd50bbff045de8adbb9d0a101ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:01:45 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:42738
I20260924 10:01:45.741050 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15772
I20260924 10:01:45.751394 16353 tablet_replica.cc:330] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: stopping tablet replica
I20260924 10:01:45.751623 16355 tablet_replica.cc:330] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: stopping tablet replica
I20260924 10:01:45.751909 16355 raft_consensus.cc:2229] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:45.751924 16353 raft_consensus.cc:2229] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:01:45.751950 16354 tablet_replica.cc:330] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: stopping tablet replica
I20260924 10:01:45.752141 16354 raft_consensus.cc:2229] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:01:45.752215 16355 raft_consensus.cc:2258] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:45.752262 16353 raft_consensus.cc:2258] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:45.752434 16354 raft_consensus.cc:2258] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:01:45.753255 16353 ts_tablet_manager.cc:1905] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:45.753677 16355 ts_tablet_manager.cc:1905] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:01:45.753877 16354 ts_tablet_manager.cc:1905] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 10:01:45 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:45.760082 16353 ts_tablet_manager.cc:1918] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:01:45.760159 16355 ts_tablet_manager.cc:1918] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:01:45.760284 16355 log.cc:1196] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/977fd50bbff045de8adbb9d0a101ab36
I20260924 10:01:45.760308 16354 ts_tablet_manager.cc:1918] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:01:45.760422 16354 log.cc:1196] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/977fd50bbff045de8adbb9d0a101ab36
I20260924 10:01:45.760500 16353 log.cc:1196] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/977fd50bbff045de8adbb9d0a101ab36
I20260924 10:01:45.760772 16355 ts_tablet_manager.cc:1939] T 977fd50bbff045de8adbb9d0a101ab36 P 8ce5e1de5fd043a5b6b99a4f49a37d7d: Deleting consensus metadata
I20260924 10:01:45.760886 16354 ts_tablet_manager.cc:1939] T 977fd50bbff045de8adbb9d0a101ab36 P 75154b9842f147a3bb0018ba890cc0be: Deleting consensus metadata
I20260924 10:01:45.760949 16353 ts_tablet_manager.cc:1939] T 977fd50bbff045de8adbb9d0a101ab36 P 7b97c26408b748c7a730bad7246ed699: Deleting consensus metadata
I20260924 10:01:45.762030 15687 catalog_manager.cc:4841] TS 75154b9842f147a3bb0018ba890cc0be (127.13.162.1:33959): tablet 977fd50bbff045de8adbb9d0a101ab36 (table test-table [id=3a1044e6dae7443385d03df1de814ae9]) successfully deleted
I20260924 10:01:45.762250 15688 catalog_manager.cc:4841] TS 8ce5e1de5fd043a5b6b99a4f49a37d7d (127.13.162.2:42345): tablet 977fd50bbff045de8adbb9d0a101ab36 (table test-table [id=3a1044e6dae7443385d03df1de814ae9]) successfully deleted
I20260924 10:01:45.762338 15687 catalog_manager.cc:4841] TS 7b97c26408b748c7a730bad7246ed699 (127.13.162.3:45049): tablet 977fd50bbff045de8adbb9d0a101ab36 (table test-table [id=3a1044e6dae7443385d03df1de814ae9]) successfully deleted
I20260924 10:01:45.766921 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:45.742845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.13.162.1:33959)
0924 10:01:45.749377 (+ 6532us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:45.749382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:45.749411 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:01:45.749728 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:45.749734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:45.749848 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:45.749954 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:45.751761 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:45.753454 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:45.755235 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:45.755881 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:45.756045 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:45.761277 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:45.761300 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:45.761322 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:45.761362 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:45.764416 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:45.765099 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:45.765108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:45.765110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:45.765250 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:45.765575 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:45.765578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:45.765580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:45.765934 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:45.766137 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:45.766398 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 10:01:45.766561 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6375,"thread_start_us":6223,"threads_started":1}
W20260924 10:01:45.767539 15903 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53940
W20260924 10:01:45.768172 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15772: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:45.768388 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15772
I20260924 10:01:45.782387 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15912
Sep 24 10:01:45 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:01:45.796522 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:45.783248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.13.162.2:42345)
0924 10:01:45.783708 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:45.783713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:45.783728 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:01:45.783778 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:45.783781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:45.783842 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:45.783940 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:45.785157 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:45.785930 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:45.787259 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:45.787675 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:45.787767 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:45.791644 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:45.791657 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:45.791673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:45.791708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:45.794384 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:45.794966 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:45.794973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:45.794976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:45.795089 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:45.795358 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:45.795360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:45.795362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:45.795693 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:45.795917 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:45.796095 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 10:01:45.796251 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
W20260924 10:01:45.797160 16041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42776
W20260924 10:01:45.797794 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:45.797923 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15912
I20260924 10:01:45.818002 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16051
Sep 24 10:01:45 dist-test-slave-jvsj krb5kdc[15656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:01:45.832266 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:45.818683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.13.162.3:45049)
0924 10:01:45.819132 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:45.819137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:45.819152 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:01:45.819216 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:45.819221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:45.819289 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:45.819397 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:45.820675 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:45.821398 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:45.822660 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:45.823046 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:45.823133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:45.827140 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:45.827155 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:45.827172 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:45.827208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:45.829952 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:45.830632 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:45.830641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:45.830645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:45.830769 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:45.831070 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:45.831075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:45.831078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:45.831441 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:45.831648 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:45.831848 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 10:01:45.832006 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":231,"threads_started":1}
W20260924 10:01:45.832875 16181 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57868
W20260924 10:01:45.833500 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:45.833657 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 16051
I20260924 10:01:45.850544 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 15669
I20260924 10:01:45.851665 15741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:45.963587 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 15669
2026-09-24T10:01:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8151 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:46 dist-test-slave-jvsj krb5kdc[16363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:46 dist-test-slave-jvsj krb5kdc[16363](info): set up 2 sockets
Sep 24 10:01:46 dist-test-slave-jvsj krb5kdc[16363](info): commencing operation
krb5kdc: starting...
W20260924 10:01:46.958205 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:47 dist-test-slave-jvsj krb5kdc[16363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:47.129552 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35801
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:47.440856 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:47.441147 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:47.441259 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:47.448614 16376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:47.448760 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:47.448923 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:47.449007 16376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:47.449065 16376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:47.459415 16376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:47.460798 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:47.462145 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:47.473539 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:47.473562 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:47.473587 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:47.474300 16376 server_base.cc:1034] running on GCE node
I20260924 10:01:47.475063 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:47.476503 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:47.477689 16376 hybrid_clock.cc:648] HybridClock initialized: now 1790244107477623 us; error 59 us; skew 500 ppm
I20260924 10:01:47.478012 16376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:47 dist-test-slave-jvsj krb5kdc[16363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244107, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:47.480985 16376 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:47.482466 16376 webserver.cc:460] Webserver started at http://127.13.162.62:43141/ using document root <none> and password file <none>
I20260924 10:01:47.483117 16376 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:47.483209 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:47.483470 16376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:47.486413 16376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "87bae9f6f9f94a31b6a2b021361dcf89"
format_stamp: "Formatted at 2026-09-24 10:01:47 on dist-test-slave-jvsj"
server_key: "f9f8e13e9f02a4933b0c74c60eac7add"
server_key_iv: "3f98e7822260eeeba3fcbd2549282d5a"
server_key_version: "encryptionkey@0"
I20260924 10:01:47.487089 16376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "87bae9f6f9f94a31b6a2b021361dcf89"
format_stamp: "Formatted at 2026-09-24 10:01:47 on dist-test-slave-jvsj"
server_key: "f9f8e13e9f02a4933b0c74c60eac7add"
server_key_iv: "3f98e7822260eeeba3fcbd2549282d5a"
server_key_version: "encryptionkey@0"
I20260924 10:01:47.492131 16376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:01:47.495610 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:47.496969 16376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:47.497115 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "87bae9f6f9f94a31b6a2b021361dcf89"
format_stamp: "Formatted at 2026-09-24 10:01:47 on dist-test-slave-jvsj"
server_key: "f9f8e13e9f02a4933b0c74c60eac7add"
server_key_iv: "3f98e7822260eeeba3fcbd2549282d5a"
server_key_version: "encryptionkey@0"
I20260924 10:01:47.497293 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:47.523698 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:47.527657 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:47.527899 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:47.545604 16376 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:35801
I20260924 10:01:47.545588 16442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:35801 every 8 connection(s)
I20260924 10:01:47.547115 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:47.550915 16443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:47.556118 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 16376
I20260924 10:01:47.556319 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:47.556912 13960 external_mini_cluster.cc:1428] Setting key d3d2cb14b5288eb911265eec248650f7
I20260924 10:01:47.561805 16443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89: Bootstrap starting.
I20260924 10:01:47.565462 16443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:47.566469 16443 log.cc:824] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:47.569078 16443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89: No bootstrap required, opened a new log
Sep 24 10:01:47 dist-test-slave-jvsj krb5kdc[16363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:47.576699 16443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35801 } }
I20260924 10:01:47.577132 16443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:47.577255 16443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bae9f6f9f94a31b6a2b021361dcf89, State: Initialized, Role: FOLLOWER
I20260924 10:01:47.577841 16443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35801 } }
I20260924 10:01:47.578087 16443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:47.578241 16443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:47.578490 16443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:47.578497 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:47.558964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.13.162.62:35801)
0924 10:01:47.559791 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:47.559804 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:47.559846 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:01:47.561586 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:47.561609 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:47.562045 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:47.562431 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:47.564159 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:47.565459 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:47.566760 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:47.567468 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:47.567642 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:47.571293 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:47.571300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:47.571318 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:47.571354 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:47.574304 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:47.574967 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:47.574973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:47.574980 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:47.575161 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:47.575525 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:47.575528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:47.575530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:47.576068 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:47.576337 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:47.576775 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 10:01:47.577192 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":384,"threads_started":1}
I20260924 10:01:47.581144 16443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35801 } }
I20260924 10:01:47.581682 16443 leader_election.cc:302] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87bae9f6f9f94a31b6a2b021361dcf89; no voters:
I20260924 10:01:47.582178 16443 leader_election.cc:288] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:47.582294 16448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:47.583041 16448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [term 1 LEADER]: Becoming Leader. State: Replica: 87bae9f6f9f94a31b6a2b021361dcf89, State: Running, Role: LEADER
I20260924 10:01:47.583611 16448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35801 } }
I20260924 10:01:47.584301 16443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:47.586272 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35801 } } }
I20260924 10:01:47.586700 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:47.587355 16450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87bae9f6f9f94a31b6a2b021361dcf89. Latest consensus state: current_term: 1 leader_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bae9f6f9f94a31b6a2b021361dcf89" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35801 } } }
I20260924 10:01:47.587749 16450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:47.588140 16457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:47.592320 16457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:47.601432 16457 catalog_manager.cc:1347] Generated new cluster ID: 67ed9ce54fad4d41820b2651c51fc5c1
I20260924 10:01:47.601537 16457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:47.615891 16457 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:47.617849 16457 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:47.638262 16457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87bae9f6f9f94a31b6a2b021361dcf89: Generated new TSK 0
I20260924 10:01:47.639477 16457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:47.691919 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35801
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:48.007659 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:48.007954 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:48.008072 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:48.015303 16469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:48.015417 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:48.015559 16469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:48.025197 16469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:48.026512 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:48.027866 16469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:48.042708 16474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:48.042783 16475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:48.042738 16477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:48.044471 16469 server_base.cc:1034] running on GCE node
I20260924 10:01:48.045042 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:48.045957 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:48.047139 16469 hybrid_clock.cc:648] HybridClock initialized: now 1790244108047080 us; error 52 us; skew 500 ppm
I20260924 10:01:48.047472 16469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:48 dist-test-slave-jvsj krb5kdc[16363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244108, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:48.050472 16469 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:48.051887 16469 webserver.cc:460] Webserver started at http://127.13.162.1:39757/ using document root <none> and password file <none>
I20260924 10:01:48.052526 16469 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:48.052621 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:48.052868 16469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:48.055696 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "e3cd925ec58d47039add4a2997a16920"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "0a88f4ececd8469de2d98059487f5459"
server_key_iv: "7780aeda6f0b96f94034d18b27800672"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.056423 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "e3cd925ec58d47039add4a2997a16920"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "0a88f4ececd8469de2d98059487f5459"
server_key_iv: "7780aeda6f0b96f94034d18b27800672"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.061676 16469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:01:48.065137 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:48.066538 16469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:48.066684 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "e3cd925ec58d47039add4a2997a16920"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "0a88f4ececd8469de2d98059487f5459"
server_key_iv: "7780aeda6f0b96f94034d18b27800672"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.066862 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:48.084542 16469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:48.088773 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:48.089025 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:48.089941 16469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:48.091281 16469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:48.091351 16469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:48.091495 16469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:48.091545 16469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:48.121013 16469 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41171
I20260924 10:01:48.121058 16596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41171 every 8 connection(s)
I20260924 10:01:48.122306 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:48.129025 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 16469
I20260924 10:01:48.129225 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:48.129786 13960 external_mini_cluster.cc:1428] Setting key 20a2dec6c6f26cb7c8f3aa7362557e73
Sep 24 10:01:48 dist-test-slave-jvsj krb5kdc[16363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244108, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:48.143006 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:48.124726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:57561 (local address 127.13.162.62:35801)
0924 10:01:48.125138 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:48.125143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:48.126120 (+ 977us) server_negotiation.cc:407] Connection header received
0924 10:01:48.127887 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:48.127892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:48.127973 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:48.128069 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:48.129586 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.130368 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.132246 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.132632 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.132724 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:48.136662 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:48.136669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:48.136674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:48.136710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:48.139902 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:48.140613 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:48.140617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:48.140621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:48.140750 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:48.141285 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:48.141289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:48.141292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:48.141604 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:48.141732 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:48.142445 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 10:01:48.142655 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":208,"threads_started":1}
I20260924 10:01:48.144030 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:48.125273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35801 (local address 127.13.162.1:57561)
0924 10:01:48.125962 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:48.125999 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:48.127632 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:48.128311 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:48.128319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:48.128812 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:48.129429 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:48.129447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.130532 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.130537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:48.132113 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:48.132128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.132847 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.132854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:48.133149 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:48.133857 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:48.133884 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:48.136419 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:48.140057 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:48.140064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:48.140078 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:48.140505 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:48.140882 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:48.140885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:48.140888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:48.141161 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:48.141730 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:48.141736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:48.142100 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 10:01:48.142369 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 10:01:48.142783 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":247,"threads_started":1}
I20260924 10:01:48.145550 16597 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35801
I20260924 10:01:48.145825 16597 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:48.146560 16597 heartbeater.cc:511] Master 127.13.162.62:35801 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:01:48.148588 16408 ts_manager.cc:194] Registered new tserver with Master: e3cd925ec58d47039add4a2997a16920 (127.13.162.1:41171)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:01:48.150079 16408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:57561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:48.168741 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35801
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:48.478583 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:48.478880 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:48.478986 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:48.486624 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:48.486747 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:48.486860 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:48.496745 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:48.498083 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:48.499372 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:48.512116 16608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:48.512122 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:48.512159 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:48.513437 16603 server_base.cc:1034] running on GCE node
I20260924 10:01:48.514022 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:48.514968 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:48.516167 16603 hybrid_clock.cc:648] HybridClock initialized: now 1790244108516125 us; error 36 us; skew 500 ppm
I20260924 10:01:48.516454 16603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:48 dist-test-slave-jvsj krb5kdc[16363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244108, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:48.519429 16603 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:48.520910 16603 webserver.cc:460] Webserver started at http://127.13.162.2:43169/ using document root <none> and password file <none>
I20260924 10:01:48.521576 16603 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:48.521695 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:48.522068 16603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:48.525306 16603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "067c0388bd454b6bafaaae8ed1accbf0"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "6fd06259300225131465b51a3a10c927"
server_key_iv: "73343fc3b0f0f4de3649e572dd563737"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.526051 16603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "067c0388bd454b6bafaaae8ed1accbf0"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "6fd06259300225131465b51a3a10c927"
server_key_iv: "73343fc3b0f0f4de3649e572dd563737"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.531224 16603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260924 10:01:48.534806 16618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:48.536330 16603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:01:48.536474 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "067c0388bd454b6bafaaae8ed1accbf0"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "6fd06259300225131465b51a3a10c927"
server_key_iv: "73343fc3b0f0f4de3649e572dd563737"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.536643 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:48.552729 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:48.556996 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:48.557253 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:48.558269 16603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:48.559651 16603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:48.559732 16603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:48.559867 16603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:48.559923 16603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:48.589718 16603 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:43097
I20260924 10:01:48.589737 16730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:43097 every 8 connection(s)
I20260924 10:01:48.591073 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:48.594805 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 16603
I20260924 10:01:48.594995 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:48.595458 13960 external_mini_cluster.cc:1428] Setting key 45fa48731a280f393e4f9f30103ae30d
Sep 24 10:01:48 dist-test-slave-jvsj krb5kdc[16363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244108, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:48.611987 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:48.593453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:45755 (local address 127.13.162.62:35801)
0924 10:01:48.593611 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:48.593616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:48.594726 (+ 1110us) server_negotiation.cc:407] Connection header received
0924 10:01:48.596892 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:48.596896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:48.596961 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:48.597035 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:48.598987 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.599726 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.601853 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.602425 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.602535 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:48.605820 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:48.605826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:48.605830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:48.605866 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:48.608976 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:48.609663 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:48.609669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:48.609672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:48.609841 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:48.610238 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:48.610243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:48.610246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:48.610608 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:48.610760 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:48.611405 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 10:01:48.611638 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:01:48.612819 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:48.593875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35801 (local address 127.13.162.2:45755)
0924 10:01:48.594574 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:48.594616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:48.596562 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:48.597316 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:48.597326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:48.598000 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:48.598812 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:48.598836 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.599867 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.599872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:48.601641 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:48.601654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:48.602633 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:48.602640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:48.602898 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:48.603548 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:48.603570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:48.605601 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:48.609119 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:48.609125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:48.609140 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:48.609544 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:48.609957 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:48.609960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:48.609963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:48.610140 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:48.610722 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:48.610728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:48.611059 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 10:01:48.611280 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 10:01:48.611674 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":266,"threads_started":1}
I20260924 10:01:48.614256 16731 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35801
I20260924 10:01:48.614602 16731 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:01:48.615442 16731 heartbeater.cc:511] Master 127.13.162.62:35801 requested a full tablet report, sending...
I20260924 10:01:48.617012 16408 ts_manager.cc:194] Registered new tserver with Master: 067c0388bd454b6bafaaae8ed1accbf0 (127.13.162.2:43097)
I20260924 10:01:48.617992 16408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:45755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:48.637144 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35801
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:48.950263 16736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:48.950559 16736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:48.950656 16736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:48.958092 16736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:48.958225 16736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:48.958384 16736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:48.967923 16736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:48.969096 16736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:48.970278 16736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:48.982496 16741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:48.982533 16742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:48.982558 16744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:48.983349 16736 server_base.cc:1034] running on GCE node
I20260924 10:01:48.983909 16736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:48.984778 16736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:48.985967 16736 hybrid_clock.cc:648] HybridClock initialized: now 1790244108985914 us; error 48 us; skew 500 ppm
I20260924 10:01:48.986243 16736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:48 dist-test-slave-jvsj krb5kdc[16363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244108, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:48.989106 16736 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:48.990438 16736 webserver.cc:460] Webserver started at http://127.13.162.3:33685/ using document root <none> and password file <none>
I20260924 10:01:48.991072 16736 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:48.991178 16736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:48.991465 16736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:48.994243 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "d9bec0b6c34b4a2bbe53667cad18c7bb"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "d327221a7f39acabcde059fba6024a22"
server_key_iv: "a012b5b73372de320d068eb9ad03a924"
server_key_version: "encryptionkey@0"
I20260924 10:01:48.995036 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "d9bec0b6c34b4a2bbe53667cad18c7bb"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "d327221a7f39acabcde059fba6024a22"
server_key_iv: "a012b5b73372de320d068eb9ad03a924"
server_key_version: "encryptionkey@0"
I20260924 10:01:49.000234 16736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 10:01:49.003800 16751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:49.005224 16736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:01:49.005403 16736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "d9bec0b6c34b4a2bbe53667cad18c7bb"
format_stamp: "Formatted at 2026-09-24 10:01:48 on dist-test-slave-jvsj"
server_key: "d327221a7f39acabcde059fba6024a22"
server_key_iv: "a012b5b73372de320d068eb9ad03a924"
server_key_version: "encryptionkey@0"
I20260924 10:01:49.005594 16736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:49.036289 16736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:49.040246 16736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:49.040493 16736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:49.041456 16736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:49.042743 16736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:49.042829 16736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:49.042945 16736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:49.043004 16736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:49.071900 16736 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:39289
I20260924 10:01:49.071981 16863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:39289 every 8 connection(s)
I20260924 10:01:49.073159 16736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:49.073578 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 16736
I20260924 10:01:49.073752 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:49.074169 13960 external_mini_cluster.cc:1428] Setting key f90d083055138681e7ca73d18c286008
Sep 24 10:01:49 dist-test-slave-jvsj krb5kdc[16363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244108, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:49.092176 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:49.075791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:36929 (local address 127.13.162.62:35801)
0924 10:01:49.075978 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:49.075985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:49.077518 (+ 1533us) server_negotiation.cc:407] Connection header received
0924 10:01:49.079315 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:49.079319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:49.079398 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:49.079476 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:49.080941 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:49.081752 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:49.083367 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:49.083784 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:49.083887 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:49.087266 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:49.087271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:49.087274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:49.087304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:49.089765 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:49.090410 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:49.090413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:49.090416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:49.090520 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:49.090881 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:49.090885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:49.090888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:49.091122 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:49.091222 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:49.091798 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 10:01:49.091951 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:01:49.093142 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:49.076474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35801 (local address 127.13.162.3:36929)
0924 10:01:49.077356 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:49.077407 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:49.079090 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:49.079714 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:49.079722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:49.080248 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:49.080811 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:49.080829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:49.081933 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:49.081937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:49.083223 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:49.083234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:49.084012 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:49.084018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:49.084272 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:49.084910 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:49.084930 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:49.087085 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:49.089905 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:49.089911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:49.089923 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:49.090285 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:49.090595 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:49.090598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:49.090601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:49.090766 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:49.091221 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:49.091227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:49.091530 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 10:01:49.091735 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 10:01:49.092121 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":414,"threads_started":1}
I20260924 10:01:49.094390 16864 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35801
I20260924 10:01:49.094661 16864 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:49.095458 16864 heartbeater.cc:511] Master 127.13.162.62:35801 requested a full tablet report, sending...
I20260924 10:01:49.096979 16408 ts_manager.cc:194] Registered new tserver with Master: d9bec0b6c34b4a2bbe53667cad18c7bb (127.13.162.3:39289)
I20260924 10:01:49.097743 16408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:36929
I20260924 10:01:49.101429 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:01:49.127554 16599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:49.125074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.13.162.62:35801)
0924 10:01:49.125273 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:49.125280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:49.125679 (+ 399us) server_negotiation.cc:407] Connection header received
0924 10:01:49.125865 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:49.125869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:49.125936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:49.126016 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:49.127290 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54444: BlockingRecv error: recv got EOF from 127.0.0.1:54444 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:01:49.130769 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16469
W20260924 10:01:49.133073 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16469: Not authorized: Client connection negotiation failed: client connection to 127.13.162.1:41171: server requires authentication, but client does not have Kerberos credentials available
I20260924 10:01:49.133255 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 16469
I20260924 10:01:49.145659 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16603
W20260924 10:01:49.148089 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16603: Not authorized: Client connection negotiation failed: client connection to 127.13.162.2:43097: server requires authentication, but client does not have Kerberos credentials available
I20260924 10:01:49.148265 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 16603
I20260924 10:01:49.160109 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16736
W20260924 10:01:49.163095 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16736: Not authorized: Client connection negotiation failed: client connection to 127.13.162.3:39289: server requires authentication, but client does not have Kerberos credentials available
I20260924 10:01:49.163220 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 16736
I20260924 10:01:49.174535 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16376
I20260924 10:01:49.175633 16438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:49.274631 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 16376
2026-09-24T10:01:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3308 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:49 dist-test-slave-jvsj krb5kdc[16881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:49 dist-test-slave-jvsj krb5kdc[16881](info): set up 2 sockets
Sep 24 10:01:49 dist-test-slave-jvsj krb5kdc[16881](info): commencing operation
krb5kdc: starting...
W20260924 10:01:50.258352 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:50 dist-test-slave-jvsj krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:50.431806 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45971
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:50.741637 16894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:50.741919 16894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:50.742046 16894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:50.749254 16894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:50.749362 16894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:50.749433 16894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:50.749511 16894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:50.749579 16894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:50.759521 16894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:45971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:50.760843 16894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:50.762161 16894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:50.773041 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:50.773102 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:50.773123 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:50.773983 16894 server_base.cc:1034] running on GCE node
I20260924 10:01:50.774747 16894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:50.776082 16894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:50.777279 16894 hybrid_clock.cc:648] HybridClock initialized: now 1790244110777232 us; error 37 us; skew 500 ppm
I20260924 10:01:50.777604 16894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:50 dist-test-slave-jvsj krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244110, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:50.786022 16894 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:50.787595 16894 webserver.cc:460] Webserver started at http://127.13.162.62:42067/ using document root <none> and password file <none>
I20260924 10:01:50.788277 16894 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:50.788385 16894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:50.788695 16894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:50.791703 16894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "ce93254a9cfa44b18846696616e07a75"
format_stamp: "Formatted at 2026-09-24 10:01:50 on dist-test-slave-jvsj"
server_key: "893a15362e1c31c012693d0f9756c5a6"
server_key_iv: "9c2a3d3dac9c7d53cc014c5a291356be"
server_key_version: "encryptionkey@0"
I20260924 10:01:50.792467 16894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "ce93254a9cfa44b18846696616e07a75"
format_stamp: "Formatted at 2026-09-24 10:01:50 on dist-test-slave-jvsj"
server_key: "893a15362e1c31c012693d0f9756c5a6"
server_key_iv: "9c2a3d3dac9c7d53cc014c5a291356be"
server_key_version: "encryptionkey@0"
I20260924 10:01:50.797458 16894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:01:50.800916 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:50.802208 16894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:50.802395 16894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "ce93254a9cfa44b18846696616e07a75"
format_stamp: "Formatted at 2026-09-24 10:01:50 on dist-test-slave-jvsj"
server_key: "893a15362e1c31c012693d0f9756c5a6"
server_key_iv: "9c2a3d3dac9c7d53cc014c5a291356be"
server_key_version: "encryptionkey@0"
I20260924 10:01:50.802587 16894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:50.827850 16894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:50.831818 16894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:50.832083 16894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:50.848534 16894 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:45971
I20260924 10:01:50.848563 16960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:45971 every 8 connection(s)
I20260924 10:01:50.849773 16894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:50.853246 16961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:50.857302 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 16894
I20260924 10:01:50.857534 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:50.858350 13960 external_mini_cluster.cc:1428] Setting key a3103f1c04361bea38431725bd7cef8c
I20260924 10:01:50.864300 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75: Bootstrap starting.
I20260924 10:01:50.867677 16961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:50.868752 16961 log.cc:824] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:50.871708 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75: No bootstrap required, opened a new log
Sep 24 10:01:50 dist-test-slave-jvsj krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:50.880311 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:50.860123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.13.162.62:45971)
0924 10:01:50.860861 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:50.860875 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:50.860922 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:01:50.862865 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:50.862889 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:50.863266 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:50.863582 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:50.865248 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:50.866595 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:50.868224 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:50.868888 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:50.869012 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:50.873377 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:50.873385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:50.873411 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:50.873450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:50.876420 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:50.877186 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:50.877192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:50.877198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:50.877352 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:50.877697 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:50.877700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:50.877702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:50.878219 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:50.878491 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:50.878895 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 10:01:50.879257 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":350,"threads_started":1}
I20260924 10:01:50.880334 16961 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93254a9cfa44b18846696616e07a75" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45971 } }
I20260924 10:01:50.880777 16961 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:50.880884 16961 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce93254a9cfa44b18846696616e07a75, State: Initialized, Role: FOLLOWER
I20260924 10:01:50.881547 16961 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93254a9cfa44b18846696616e07a75" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45971 } }
I20260924 10:01:50.881803 16961 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:50.881946 16961 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:50.882102 16961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:50.884181 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:50.884648 16961 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93254a9cfa44b18846696616e07a75" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45971 } }
I20260924 10:01:50.885095 16961 leader_election.cc:302] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce93254a9cfa44b18846696616e07a75; no voters:
I20260924 10:01:50.885643 16961 leader_election.cc:288] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:50.885743 16966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:50.887859 16961 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:50.892541 16966 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [term 1 LEADER]: Becoming Leader. State: Replica: ce93254a9cfa44b18846696616e07a75, State: Running, Role: LEADER
I20260924 10:01:50.893147 16966 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93254a9cfa44b18846696616e07a75" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45971 } }
I20260924 10:01:50.899582 16968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce93254a9cfa44b18846696616e07a75. Latest consensus state: current_term: 1 leader_uuid: "ce93254a9cfa44b18846696616e07a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93254a9cfa44b18846696616e07a75" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45971 } } }
I20260924 10:01:50.899956 16968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:50.900484 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce93254a9cfa44b18846696616e07a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93254a9cfa44b18846696616e07a75" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45971 } } }
I20260924 10:01:50.900748 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:50.902786 16981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:50.910987 16981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:50.924444 16981 catalog_manager.cc:1347] Generated new cluster ID: 93f44e5a319046c9a301b3b893bd9375
I20260924 10:01:50.924553 16981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:50.931008 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:50.917650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.13.162.62:45971)
0924 10:01:50.917838 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:50.917845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:50.918367 (+ 522us) server_negotiation.cc:407] Connection header received
0924 10:01:50.918594 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:50.918599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:50.918661 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:50.918740 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:50.919682 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:50.920387 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:50.921617 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:50.922041 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:50.922145 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:50.923285 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:50.923291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:50.923296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:50.924074 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:50.924147 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:50.924182 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:50.924303 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:50.930563 (+ 6260us) server_negotiation.cc:299] Negotiation successful
0924 10:01:50.930660 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:01:50.934675 16981 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:50.935917 16981 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:50.945508 16981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce93254a9cfa44b18846696616e07a75: Generated new TSK 0
I20260924 10:01:50.946497 16981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:50.956383 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 16894
I20260924 10:01:50.957671 16956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:51.043725 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 16894
2026-09-24T10:01:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1763 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:51 dist-test-slave-jvsj krb5kdc[16996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:51 dist-test-slave-jvsj krb5kdc[16996](info): set up 2 sockets
Sep 24 10:01:51 dist-test-slave-jvsj krb5kdc[16996](info): commencing operation
krb5kdc: starting...
W20260924 10:01:52.022362 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:52 dist-test-slave-jvsj krb5kdc[16996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:52.191519 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:39391
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:52.498685 17009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:52.499003 17009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:52.499140 17009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:52.506469 17009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:52.506590 17009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:52.506669 17009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:52.506747 17009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:52.506806 17009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:52.516849 17009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:52.518209 17009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:52.519524 17009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:52.530498 17017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:52.530562 17014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:52.530582 17015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:52.531618 17009 server_base.cc:1034] running on GCE node
I20260924 10:01:52.532320 17009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:52.533653 17009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:52.534874 17009 hybrid_clock.cc:648] HybridClock initialized: now 1790244112534817 us; error 43 us; skew 500 ppm
I20260924 10:01:52.535207 17009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:52 dist-test-slave-jvsj krb5kdc[16996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244112, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:52.538194 17009 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:52.539623 17009 webserver.cc:460] Webserver started at http://127.13.162.62:36651/ using document root <none> and password file <none>
I20260924 10:01:52.540220 17009 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:52.540313 17009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:52.540563 17009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:52.543437 17009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "523c8343514f42c198481c3f22e62e30"
format_stamp: "Formatted at 2026-09-24 10:01:52 on dist-test-slave-jvsj"
server_key: "57faec6d586d087fb747f1613f404023"
server_key_iv: "773eabd58ae9e169fe78063176987b83"
server_key_version: "encryptionkey@0"
I20260924 10:01:52.544144 17009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "523c8343514f42c198481c3f22e62e30"
format_stamp: "Formatted at 2026-09-24 10:01:52 on dist-test-slave-jvsj"
server_key: "57faec6d586d087fb747f1613f404023"
server_key_iv: "773eabd58ae9e169fe78063176987b83"
server_key_version: "encryptionkey@0"
I20260924 10:01:52.549147 17009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 10:01:52.552448 17024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:52.553715 17009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:01:52.553905 17009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "523c8343514f42c198481c3f22e62e30"
format_stamp: "Formatted at 2026-09-24 10:01:52 on dist-test-slave-jvsj"
server_key: "57faec6d586d087fb747f1613f404023"
server_key_iv: "773eabd58ae9e169fe78063176987b83"
server_key_version: "encryptionkey@0"
I20260924 10:01:52.554070 17009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:52.572893 17009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:52.576592 17009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:52.576859 17009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:52.593140 17009 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:39391
I20260924 10:01:52.593159 17075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:39391 every 8 connection(s)
I20260924 10:01:52.594568 17009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:52.596979 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17009
I20260924 10:01:52.597162 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:52.597774 13960 external_mini_cluster.cc:1428] Setting key 7dd0c647724722559d6ddb4b156a6a09
I20260924 10:01:52.598245 17076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:52.611660 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30: Bootstrap starting.
Sep 24 10:01:52 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:52.614655 17076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:52.615655 17076 log.cc:824] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:52.618257 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30: No bootstrap required, opened a new log
I20260924 10:01:52.619689 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:52.599677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.13.162.62:39391)
0924 10:01:52.600412 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:52.600423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:52.600461 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:01:52.602612 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:52.602633 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:52.603027 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:52.603393 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:52.604955 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:52.606212 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:52.607880 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:52.608581 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:52.608725 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:52.613109 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:52.613118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:52.613133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:52.613177 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:52.616008 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:52.616632 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:52.616642 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:52.616649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:52.616818 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:52.617171 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:52.617175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:52.617177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:52.617661 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:52.617944 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:52.618352 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 10:01:52.618740 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":326,"threads_started":1}
I20260924 10:01:52.625795 17076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c8343514f42c198481c3f22e62e30" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39391 } }
I20260924 10:01:52.626130 17076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:52.626215 17076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523c8343514f42c198481c3f22e62e30, State: Initialized, Role: FOLLOWER
I20260924 10:01:52.626742 17076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c8343514f42c198481c3f22e62e30" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39391 } }
I20260924 10:01:52.626958 17076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:52.627074 17076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:52.627211 17076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:52.629459 17076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c8343514f42c198481c3f22e62e30" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39391 } }
I20260924 10:01:52.629886 17076 leader_election.cc:302] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523c8343514f42c198481c3f22e62e30; no voters:
I20260924 10:01:52.630440 17076 leader_election.cc:288] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:52.630555 17081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:52.631125 17081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [term 1 LEADER]: Becoming Leader. State: Replica: 523c8343514f42c198481c3f22e62e30, State: Running, Role: LEADER
I20260924 10:01:52.631632 17081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c8343514f42c198481c3f22e62e30" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39391 } }
I20260924 10:01:52.632316 17076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:52.634501 17082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523c8343514f42c198481c3f22e62e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c8343514f42c198481c3f22e62e30" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39391 } } }
I20260924 10:01:52.634577 17083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523c8343514f42c198481c3f22e62e30. Latest consensus state: current_term: 1 leader_uuid: "523c8343514f42c198481c3f22e62e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c8343514f42c198481c3f22e62e30" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 39391 } } }
I20260924 10:01:52.634864 17082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:52.634923 17083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:52.635629 17089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:52.640332 17089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:52.648663 17089 catalog_manager.cc:1347] Generated new cluster ID: c9188301dfd64273b32745344a998b61
I20260924 10:01:52.648774 17089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:52.692126 17089 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:52.693423 17089 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:52.700011 17089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523c8343514f42c198481c3f22e62e30: Generated new TSK 0
I20260924 10:01:52.701076 17089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:52.751384 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39391
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:53.056733 17102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:53.057041 17102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:53.057188 17102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:53.064312 17102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:53.064422 17102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:53.064569 17102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:53.074393 17102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:53.075664 17102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:53.076907 17102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:53.088989 17107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:53.089022 17108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:53.089056 17110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:53.089859 17102 server_base.cc:1034] running on GCE node
I20260924 10:01:53.090451 17102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:53.091305 17102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:53.092499 17102 hybrid_clock.cc:648] HybridClock initialized: now 1790244113092454 us; error 52 us; skew 500 ppm
I20260924 10:01:53.092784 17102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:53 dist-test-slave-jvsj krb5kdc[16996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244113, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:53.095731 17102 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:53.097090 17102 webserver.cc:460] Webserver started at http://127.13.162.1:43547/ using document root <none> and password file <none>
I20260924 10:01:53.097721 17102 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:53.097831 17102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:53.098099 17102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:53.100852 17102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "e08ea6c2754c4e42a2f8655961dee4d3"
format_stamp: "Formatted at 2026-09-24 10:01:53 on dist-test-slave-jvsj"
server_key: "c2d040d69c24b965954d90074b127a37"
server_key_iv: "e0b0a75a4f15b37b893dc77c3cae7134"
server_key_version: "encryptionkey@0"
I20260924 10:01:53.101572 17102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "e08ea6c2754c4e42a2f8655961dee4d3"
format_stamp: "Formatted at 2026-09-24 10:01:53 on dist-test-slave-jvsj"
server_key: "c2d040d69c24b965954d90074b127a37"
server_key_iv: "e0b0a75a4f15b37b893dc77c3cae7134"
server_key_version: "encryptionkey@0"
I20260924 10:01:53.106251 17102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 10:01:53.109388 17117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:53.110867 17102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:53.111022 17102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "e08ea6c2754c4e42a2f8655961dee4d3"
format_stamp: "Formatted at 2026-09-24 10:01:53 on dist-test-slave-jvsj"
server_key: "c2d040d69c24b965954d90074b127a37"
server_key_iv: "e0b0a75a4f15b37b893dc77c3cae7134"
server_key_version: "encryptionkey@0"
I20260924 10:01:53.111194 17102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:53.133313 17102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:53.137192 17102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:53.137447 17102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:53.138363 17102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:53.139613 17102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:53.139690 17102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:53.139807 17102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:53.139858 17102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:53.168015 17102 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:44965
I20260924 10:01:53.168088 17229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:44965 every 8 connection(s)
I20260924 10:01:53.169221 17102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:53.177388 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17102
I20260924 10:01:53.177568 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:53.178153 13960 external_mini_cluster.cc:1428] Setting key e8fa6afcb60e934fbf67ba2d6138501d
Sep 24 10:01:53 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244113, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:53.188246 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:53.171576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:56051 (local address 127.13.162.62:39391)
0924 10:01:53.171984 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:53.171988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:53.172948 (+ 960us) server_negotiation.cc:407] Connection header received
0924 10:01:53.174663 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:53.174668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:53.174735 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:53.174832 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:53.176246 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.177052 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.178687 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.179200 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.179305 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:53.182766 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:53.182771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:53.182774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:53.182806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:53.185229 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:53.186045 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:53.186050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:53.186052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:53.186163 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:53.186610 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:53.186614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:53.186616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:53.186938 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:53.187049 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:53.187838 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 10:01:53.187997 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":199,"threads_started":1}
I20260924 10:01:53.189595 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:53.172121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39391 (local address 127.13.162.1:56051)
0924 10:01:53.172767 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:53.172839 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:53.174400 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:53.175061 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:53.175070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:53.175545 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:53.176098 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:53.176114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.177203 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.177207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:53.178558 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:53.178570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.179411 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.179419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:53.179649 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:53.180301 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:53.180327 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:53.182556 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:53.185358 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:53.185368 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:53.185386 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:53.185931 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:53.186250 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:53.186255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:53.186259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:53.186492 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:53.187050 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:53.187057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:53.187456 (+ 399us) client_negotiation.cc:769] Sending connection context
0924 10:01:53.187726 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 10:01:53.188246 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":247,"threads_started":1}
I20260924 10:01:53.191165 17230 heartbeater.cc:347] Connected to a master server at 127.13.162.62:39391
I20260924 10:01:53.191448 17230 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:53.192114 17230 heartbeater.cc:511] Master 127.13.162.62:39391 requested a full tablet report, sending...
I20260924 10:01:53.194096 17041 ts_manager.cc:194] Registered new tserver with Master: e08ea6c2754c4e42a2f8655961dee4d3 (127.13.162.1:44965)
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:01:53.195672 17041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:56051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:53.217926 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39391
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:53.526692 17236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:53.527143 17236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:53.527287 17236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:53.534646 17236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:53.534761 17236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:53.534924 17236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:53.544968 17236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:53.546239 17236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:53.547657 17236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:53.560058 17241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:53.560029 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:53.559983 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:53.560868 17236 server_base.cc:1034] running on GCE node
I20260924 10:01:53.561468 17236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:53.562498 17236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:53.563690 17236 hybrid_clock.cc:648] HybridClock initialized: now 1790244113563632 us; error 56 us; skew 500 ppm
I20260924 10:01:53.563995 17236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:53 dist-test-slave-jvsj krb5kdc[16996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244113, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:53.569840 17236 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:53.571331 17236 webserver.cc:460] Webserver started at http://127.13.162.2:37001/ using document root <none> and password file <none>
I20260924 10:01:53.572021 17236 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:53.572126 17236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:53.572408 17236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:53.575350 17236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "6b6d56f1b6214e4dbe0c896d964d5488"
format_stamp: "Formatted at 2026-09-24 10:01:53 on dist-test-slave-jvsj"
server_key: "80a7533f4d436420b14f747b8dc45958"
server_key_iv: "1afa459744de8ae976725397fec042d2"
server_key_version: "encryptionkey@0"
I20260924 10:01:53.576103 17236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "6b6d56f1b6214e4dbe0c896d964d5488"
format_stamp: "Formatted at 2026-09-24 10:01:53 on dist-test-slave-jvsj"
server_key: "80a7533f4d436420b14f747b8dc45958"
server_key_iv: "1afa459744de8ae976725397fec042d2"
server_key_version: "encryptionkey@0"
I20260924 10:01:53.581130 17236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:01:53.584506 17251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:53.585811 17236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:01:53.585952 17236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "6b6d56f1b6214e4dbe0c896d964d5488"
format_stamp: "Formatted at 2026-09-24 10:01:53 on dist-test-slave-jvsj"
server_key: "80a7533f4d436420b14f747b8dc45958"
server_key_iv: "1afa459744de8ae976725397fec042d2"
server_key_version: "encryptionkey@0"
I20260924 10:01:53.586122 17236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:53.618221 17236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:53.623620 17236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:53.623904 17236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:53.625077 17236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:53.626900 17236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:53.626992 17236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:53.627113 17236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:53.627170 17236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:53.660322 17236 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:39435
I20260924 10:01:53.660425 17363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:39435 every 8 connection(s)
I20260924 10:01:53.661564 17236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:53.664321 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17236
I20260924 10:01:53.664497 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:53.665045 13960 external_mini_cluster.cc:1428] Setting key aa8d791567694e0a9b655e51a7ee7372
Sep 24 10:01:53 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244113, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:01:53.683023 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:53.664293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:59897 (local address 127.13.162.62:39391)
0924 10:01:53.664484 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:53.664490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:53.665982 (+ 1492us) server_negotiation.cc:407] Connection header received
0924 10:01:53.668376 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:53.668382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:53.668476 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:53.668566 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:53.670093 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.670974 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.673051 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.673471 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.673589 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:53.676978 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:53.676984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:53.676988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:53.677019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:53.680211 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:53.680914 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:53.680920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:53.680924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:53.681077 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:53.681451 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:53.681456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:53.681460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:53.681775 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:53.681895 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:53.682573 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 10:01:53.682743 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:01:53.684059 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:53.664951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39391 (local address 127.13.162.2:59897)
0924 10:01:53.665810 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:53.665867 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:53.668105 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:53.668839 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:53.668848 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:53.669339 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:53.669931 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:53.669949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.671154 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.671159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:53.672915 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:53.672931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:53.673684 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:53.673690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:53.673933 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:53.674634 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:53.674657 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:53.676679 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:53.680359 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:53.680366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:53.680380 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:53.680796 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:53.681175 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:53.681178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:53.681180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:53.681349 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:53.681877 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:53.681883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:53.682199 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 10:01:53.682487 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 10:01:53.682868 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":315,"threads_started":1}
I20260924 10:01:53.685470 17364 heartbeater.cc:347] Connected to a master server at 127.13.162.62:39391
I20260924 10:01:53.685782 17364 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:53.686652 17364 heartbeater.cc:511] Master 127.13.162.62:39391 requested a full tablet report, sending...
I20260924 10:01:53.688205 17041 ts_manager.cc:194] Registered new tserver with Master: 6b6d56f1b6214e4dbe0c896d964d5488 (127.13.162.2:39435)
I20260924 10:01:53.689049 17041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:59897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:53.705193 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39391
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:54.009200 17369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:54.009457 17369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:54.009574 17369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:54.016666 17369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:54.016790 17369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:54.016960 17369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:54.026392 17369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:39391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:54.027611 17369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:54.028843 17369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:54.041368 17374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:54.041388 17375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:54.041985 17377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:54.042068 17369 server_base.cc:1034] running on GCE node
I20260924 10:01:54.042670 17369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:54.043501 17369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:54.044692 17369 hybrid_clock.cc:648] HybridClock initialized: now 1790244114044656 us; error 32 us; skew 500 ppm
I20260924 10:01:54.044963 17369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:54.047874 17369 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:54.049211 17369 webserver.cc:460] Webserver started at http://127.13.162.3:42783/ using document root <none> and password file <none>
I20260924 10:01:54.049846 17369 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:54.049948 17369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:54.050235 17369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:54.053038 17369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "66349be4bb9e4ea1af592d87eff6098c"
format_stamp: "Formatted at 2026-09-24 10:01:54 on dist-test-slave-jvsj"
server_key: "e5b2b93271e4bf861a9ca0e250c5e4da"
server_key_iv: "90482985609237f783585bf084995614"
server_key_version: "encryptionkey@0"
I20260924 10:01:54.053757 17369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "66349be4bb9e4ea1af592d87eff6098c"
format_stamp: "Formatted at 2026-09-24 10:01:54 on dist-test-slave-jvsj"
server_key: "e5b2b93271e4bf861a9ca0e250c5e4da"
server_key_iv: "90482985609237f783585bf084995614"
server_key_version: "encryptionkey@0"
I20260924 10:01:54.058713 17369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 10:01:54.062023 17384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:54.063401 17369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:54.063540 17369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "66349be4bb9e4ea1af592d87eff6098c"
format_stamp: "Formatted at 2026-09-24 10:01:54 on dist-test-slave-jvsj"
server_key: "e5b2b93271e4bf861a9ca0e250c5e4da"
server_key_iv: "90482985609237f783585bf084995614"
server_key_version: "encryptionkey@0"
I20260924 10:01:54.063709 17369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:54.098194 17369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:54.102162 17369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:54.102443 17369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:54.103338 17369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:54.104570 17369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:54.104638 17369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:54.104753 17369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:54.104799 17369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:54.132647 17369 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:38465
I20260924 10:01:54.132716 17496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:38465 every 8 connection(s)
I20260924 10:01:54.133876 17369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:54.140611 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17369
I20260924 10:01:54.140746 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:54.141160 13960 external_mini_cluster.cc:1428] Setting key cf9893185bce95ac30b68ac87aefcef0
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:54.152223 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:54.136233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:38269 (local address 127.13.162.62:39391)
0924 10:01:54.136396 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:54.136400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:54.137578 (+ 1178us) server_negotiation.cc:407] Connection header received
0924 10:01:54.139443 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:54.139448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:54.139541 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:54.139626 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:54.141095 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.142011 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.143639 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.144115 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.144210 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:54.147296 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:54.147301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:54.147304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:54.147349 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:54.149695 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.150361 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.150366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.150370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.150485 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.150831 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.150834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.150836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.151086 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:54.151190 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:54.151826 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 10:01:54.151988 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:01:54.153172 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:54.136694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:39391 (local address 127.13.162.3:38269)
0924 10:01:54.137427 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:54.137464 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:54.139178 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:54.139865 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:54.139874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:54.140353 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:54.140940 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:54.140959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.142145 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.142148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:54.143517 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:54.143529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.144298 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.144304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:54.144496 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:54.145070 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:54.145097 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:54.147122 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:54.149816 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:54.149823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:54.149836 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:54.150222 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:54.150568 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:54.150571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:54.150573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:54.150734 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:54.151182 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:54.151187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:54.151488 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 10:01:54.151709 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 10:01:54.152073 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":300,"threads_started":1}
I20260924 10:01:54.154592 17497 heartbeater.cc:347] Connected to a master server at 127.13.162.62:39391
I20260924 10:01:54.154892 17497 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:54.155704 17497 heartbeater.cc:511] Master 127.13.162.62:39391 requested a full tablet report, sending...
I20260924 10:01:54.157004 17041 ts_manager.cc:194] Registered new tserver with Master: 66349be4bb9e4ea1af592d87eff6098c (127.13.162.3:38465)
I20260924 10:01:54.157826 17041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:38269
I20260924 10:01:54.168218 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260924 10:01:54.199209 17230 heartbeater.cc:503] Master 127.13.162.62:39391 was elected leader, sending a full tablet report...
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:54.213167 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:54.199498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.13.162.62:39391)
0924 10:01:54.199673 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:54.199679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:54.200167 (+ 488us) server_negotiation.cc:407] Connection header received
0924 10:01:54.200340 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:54.200345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:54.200411 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:54.200489 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:54.201842 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.202643 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.203809 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.204221 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.204327 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:54.207956 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:54.207961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:54.207964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:54.207994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:54.210196 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.210878 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.210883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.210887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.211030 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.211465 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.211470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.211473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.211903 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:54.212107 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:54.212625 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 10:01:54.212839 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 10:01:54.217639 17041 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33080
I20260924 10:01:54.219400 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17102
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:54.233894 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:54.220741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.13.162.1:44965)
0924 10:01:54.221331 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:54.221338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:54.221362 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:01:54.221411 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:54.221419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:54.221621 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:54.221746 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:54.222847 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.223855 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.224950 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.225494 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.225584 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:54.229050 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:54.229060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:54.229074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:54.229106 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:54.231605 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.232211 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.232218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.232220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.232348 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.232639 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.232642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.232644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.232947 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:54.233172 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:54.233428 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 10:01:54.233628 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":306,"threads_started":1}
W20260924 10:01:54.235148 17225 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53506
W20260924 10:01:54.236266 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17102: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:54.236461 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17102
I20260924 10:01:54.248687 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17236
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:01:54.263701 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:54.249533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.13.162.2:39435)
0924 10:01:54.250115 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:54.250122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:54.250149 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:01:54.250207 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:54.250213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:54.250465 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:54.250593 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:54.251753 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.252757 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.253950 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.254587 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.254693 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:54.258437 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:54.258448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:54.258463 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:54.258495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:54.260998 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.261661 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.261669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.261671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.261834 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.262214 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.262217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.262220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.262693 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:54.262956 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:54.263141 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 10:01:54.263352 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":329,"threads_started":1}
W20260924 10:01:54.264955 17359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51156
W20260924 10:01:54.266072 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17236: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:54.266211 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17236
I20260924 10:01:54.278357 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17369
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[16996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244114, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:01:54.293862 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:54.279268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.13.162.3:38465)
0924 10:01:54.279870 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:54.279877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:54.279906 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:01:54.279963 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:54.279970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:54.280180 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:54.280308 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:54.281553 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.282692 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.283887 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:54.284460 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:54.284555 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:54.288905 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:54.288917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:54.288932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:54.288972 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:54.291518 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.292129 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.292136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.292138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.292256 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:54.292567 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:54.292571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:54.292573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:54.292934 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:54.293172 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:54.293376 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 10:01:54.293581 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":292,"threads_started":1}
W20260924 10:01:54.295166 17492 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52122
W20260924 10:01:54.296213 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17369: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:54.296357 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17369
I20260924 10:01:54.308079 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17009
I20260924 10:01:54.309186 17071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:54.409144 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17009
2026-09-24T10:01:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3370 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[17514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[17514](info): set up 2 sockets
Sep 24 10:01:54 dist-test-slave-jvsj krb5kdc[17514](info): commencing operation
krb5kdc: starting...
W20260924 10:01:55.410378 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:55 dist-test-slave-jvsj krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:55.585566 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42217
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:55.889535 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:55.889853 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:55.889983 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:55.897460 17527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:55.897573 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:55.897648 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:55.897722 17527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:55.897779 17527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:55.907503 17527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:42217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:55.908761 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:55.910024 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:55.921188 17532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:55.921197 17533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:55.921095 17535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:55.922130 17527 server_base.cc:1034] running on GCE node
I20260924 10:01:55.922919 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:55.924304 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:55.925513 17527 hybrid_clock.cc:648] HybridClock initialized: now 1790244115925460 us; error 42 us; skew 500 ppm
I20260924 10:01:55.925812 17527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:55 dist-test-slave-jvsj krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244115, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:55.932235 17527 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:55.933663 17527 webserver.cc:460] Webserver started at http://127.13.162.62:36223/ using document root <none> and password file <none>
I20260924 10:01:55.934345 17527 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:55.934456 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:55.934777 17527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:55.937754 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "d0692b84d313481ca3ec9ad437b55ca4"
format_stamp: "Formatted at 2026-09-24 10:01:55 on dist-test-slave-jvsj"
server_key: "96e3f35f742b198d7c0f4bad9f0fa9e6"
server_key_iv: "ac295c878f742ea9c054b3e305668038"
server_key_version: "encryptionkey@0"
I20260924 10:01:55.938578 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "d0692b84d313481ca3ec9ad437b55ca4"
format_stamp: "Formatted at 2026-09-24 10:01:55 on dist-test-slave-jvsj"
server_key: "96e3f35f742b198d7c0f4bad9f0fa9e6"
server_key_iv: "ac295c878f742ea9c054b3e305668038"
server_key_version: "encryptionkey@0"
I20260924 10:01:55.943677 17527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:01:55.947105 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:55.948411 17527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:55.948563 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "d0692b84d313481ca3ec9ad437b55ca4"
format_stamp: "Formatted at 2026-09-24 10:01:55 on dist-test-slave-jvsj"
server_key: "96e3f35f742b198d7c0f4bad9f0fa9e6"
server_key_iv: "ac295c878f742ea9c054b3e305668038"
server_key_version: "encryptionkey@0"
I20260924 10:01:55.948738 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:55.964500 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:55.968375 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:55.968623 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:55.985251 17527 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:42217
I20260924 10:01:55.985261 17593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:42217 every 8 connection(s)
I20260924 10:01:55.986642 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:55.990175 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:55.991823 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17527
I20260924 10:01:55.992039 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:55.992781 13960 external_mini_cluster.cc:1428] Setting key bcc9d9755e0133a756256187b52583cc
I20260924 10:01:56.001497 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4: Bootstrap starting.
I20260924 10:01:56.004470 17594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:56.005427 17594 log.cc:824] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:01:56 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:56.008097 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4: No bootstrap required, opened a new log
I20260924 10:01:56.013480 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:55.994910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.13.162.62:42217)
0924 10:01:55.995640 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:55.995656 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:55.995697 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:01:55.997391 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:55.997412 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:55.997799 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:55.998148 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:55.999564 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:56.000876 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:56.002070 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:56.002791 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:56.002985 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:56.006679 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:56.006686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:56.006702 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:56.006736 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:56.009551 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:56.010199 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:56.010206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:56.010212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:56.010435 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:56.010804 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:56.010808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:56.010810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:56.011352 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:56.011638 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:56.012059 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 10:01:56.012455 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":302,"threads_started":1}
I20260924 10:01:56.016108 17594 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0692b84d313481ca3ec9ad437b55ca4" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42217 } }
I20260924 10:01:56.016568 17594 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:56.016664 17594 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0692b84d313481ca3ec9ad437b55ca4, State: Initialized, Role: FOLLOWER
I20260924 10:01:56.017192 17594 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0692b84d313481ca3ec9ad437b55ca4" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42217 } }
I20260924 10:01:56.017402 17594 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:56.017521 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:56.017660 17594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:56.019732 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0692b84d313481ca3ec9ad437b55ca4" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42217 } }
I20260924 10:01:56.020331 17594 leader_election.cc:302] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0692b84d313481ca3ec9ad437b55ca4; no voters:
I20260924 10:01:56.021023 17594 leader_election.cc:288] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:56.021098 17599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:56.021744 17599 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [term 1 LEADER]: Becoming Leader. State: Replica: d0692b84d313481ca3ec9ad437b55ca4, State: Running, Role: LEADER
I20260924 10:01:56.022254 17599 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0692b84d313481ca3ec9ad437b55ca4" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42217 } }
I20260924 10:01:56.022949 17594 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:56.025000 17601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0692b84d313481ca3ec9ad437b55ca4. Latest consensus state: current_term: 1 leader_uuid: "d0692b84d313481ca3ec9ad437b55ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0692b84d313481ca3ec9ad437b55ca4" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42217 } } }
I20260924 10:01:56.025350 17601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:56.025799 17600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0692b84d313481ca3ec9ad437b55ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0692b84d313481ca3ec9ad437b55ca4" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42217 } } }
I20260924 10:01:56.026084 17600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:56.026759 17604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:56.031513 17604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:56.040184 17604 catalog_manager.cc:1347] Generated new cluster ID: 22af54f534a9423e867a36ecf3b4a2e2
I20260924 10:01:56.040304 17604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:56.061782 17604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:56.063107 17604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:56.072016 17604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0692b84d313481ca3ec9ad437b55ca4: Generated new TSK 0
I20260924 10:01:56.072844 17604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:56.125933 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42217
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:56.444450 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:56.444746 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:56.444877 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:56.452050 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:56.452155 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:56.452313 17620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:01:56.462000 17620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:56.463343 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:56.464680 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:56.478673 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:56.478734 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:56.478705 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:56.479926 17620 server_base.cc:1034] running on GCE node
I20260924 10:01:56.480533 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:56.481478 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:56.482690 17620 hybrid_clock.cc:648] HybridClock initialized: now 1790244116482628 us; error 48 us; skew 500 ppm
I20260924 10:01:56.483022 17620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:56 dist-test-slave-jvsj krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:56.486075 17620 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:01:56.487633 17620 webserver.cc:460] Webserver started at http://127.13.162.1:45267/ using document root <none> and password file <none>
I20260924 10:01:56.488324 17620 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:56.488442 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:56.488727 17620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:56.491696 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "b631e3b3f0514f2ca9c9b1f62bc4ed03"
format_stamp: "Formatted at 2026-09-24 10:01:56 on dist-test-slave-jvsj"
server_key: "24d5ca390d992900b0fd02d062d4b5b8"
server_key_iv: "a38611b0b70d6dd8d5e0bac9d6babb69"
server_key_version: "encryptionkey@0"
I20260924 10:01:56.492452 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "b631e3b3f0514f2ca9c9b1f62bc4ed03"
format_stamp: "Formatted at 2026-09-24 10:01:56 on dist-test-slave-jvsj"
server_key: "24d5ca390d992900b0fd02d062d4b5b8"
server_key_iv: "a38611b0b70d6dd8d5e0bac9d6babb69"
server_key_version: "encryptionkey@0"
I20260924 10:01:56.497594 17620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:01:56.501142 17635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:56.502549 17620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:56.502693 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "b631e3b3f0514f2ca9c9b1f62bc4ed03"
format_stamp: "Formatted at 2026-09-24 10:01:56 on dist-test-slave-jvsj"
server_key: "24d5ca390d992900b0fd02d062d4b5b8"
server_key_iv: "a38611b0b70d6dd8d5e0bac9d6babb69"
server_key_version: "encryptionkey@0"
I20260924 10:01:56.502873 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:56.530644 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:56.535087 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:56.535355 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:56.536423 17620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:56.537845 17620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:56.537933 17620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:56.538056 17620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:56.538110 17620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:56.567327 17620 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:35795
I20260924 10:01:56.567370 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:35795 every 8 connection(s)
I20260924 10:01:56.568544 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:01:56.575546 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17620
I20260924 10:01:56.575759 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:01:56.576421 13960 external_mini_cluster.cc:1428] Setting key 0effe01327b3032a9ad728fa48fe9f92
Sep 24 10:01:56 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:56.588637 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:56.570898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:46363 (local address 127.13.162.62:42217)
0924 10:01:56.571270 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:56.571274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:56.572354 (+ 1080us) server_negotiation.cc:407] Connection header received
0924 10:01:56.574119 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:56.574124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:56.574197 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:56.574286 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:56.576031 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:56.576758 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:56.578614 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:56.579084 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:56.579178 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:56.583094 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:56.583099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:56.583102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:56.583132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:56.585850 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:56.586571 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:56.586576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:56.586580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:56.586709 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:56.587070 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:56.587075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:56.587078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:56.587377 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:56.587521 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:56.588124 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 10:01:56.588322 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":195,"threads_started":1}
I20260924 10:01:56.589463 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:56.571475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42217 (local address 127.13.162.1:46363)
0924 10:01:56.572202 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:56.572248 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:56.573845 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:56.574542 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:56.574550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:56.575039 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:56.575857 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:56.575884 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:56.576909 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:56.576914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:56.578476 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:56.578490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:56.579291 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:56.579297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:56.579592 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:56.580282 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:56.580310 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:56.582878 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:56.585977 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:56.585984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:56.585997 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:56.586456 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:56.586802 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:56.586805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:56.586807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:56.586980 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:56.587482 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:56.587488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:56.587782 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 10:01:56.587992 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 10:01:56.588357 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":246,"threads_started":1}
I20260924 10:01:56.591128 17748 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42217
I20260924 10:01:56.591403 17748 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:56.592072 17748 heartbeater.cc:511] Master 127.13.162.62:42217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:01:56.594049 17559 ts_manager.cc:194] Registered new tserver with Master: b631e3b3f0514f2ca9c9b1f62bc4ed03 (127.13.162.1:35795)
I20260924 10:01:56.595430 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:46363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:01:56.617735 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42217
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:56.928478 17754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:56.928773 17754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:56.928913 17754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:56.936074 17754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:56.936187 17754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:56.936338 17754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:01:56.945791 17754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:56.947037 17754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:56.948277 17754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:56.960698 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:56.960734 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:56.960774 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:56.961555 17754 server_base.cc:1034] running on GCE node
I20260924 10:01:56.962106 17754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:56.963034 17754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:56.964229 17754 hybrid_clock.cc:648] HybridClock initialized: now 1790244116964169 us; error 50 us; skew 500 ppm
I20260924 10:01:56.964531 17754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:56 dist-test-slave-jvsj krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:56.967432 17754 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:01:56.968945 17754 webserver.cc:460] Webserver started at http://127.13.162.2:36561/ using document root <none> and password file <none>
I20260924 10:01:56.969591 17754 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:56.969702 17754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:56.969983 17754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:56.972779 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "1488b112762e45998da1623755130c29"
format_stamp: "Formatted at 2026-09-24 10:01:56 on dist-test-slave-jvsj"
server_key: "5d03af3219638f3af58c584236627e79"
server_key_iv: "103ef58aee9570e87bf7c19dd58a2bf0"
server_key_version: "encryptionkey@0"
I20260924 10:01:56.973500 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "1488b112762e45998da1623755130c29"
format_stamp: "Formatted at 2026-09-24 10:01:56 on dist-test-slave-jvsj"
server_key: "5d03af3219638f3af58c584236627e79"
server_key_iv: "103ef58aee9570e87bf7c19dd58a2bf0"
server_key_version: "encryptionkey@0"
I20260924 10:01:56.978308 17754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 10:01:56.981808 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:56.983203 17754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:56.983350 17754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "1488b112762e45998da1623755130c29"
format_stamp: "Formatted at 2026-09-24 10:01:56 on dist-test-slave-jvsj"
server_key: "5d03af3219638f3af58c584236627e79"
server_key_iv: "103ef58aee9570e87bf7c19dd58a2bf0"
server_key_version: "encryptionkey@0"
I20260924 10:01:56.983531 17754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:56.995422 17754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:56.999392 17754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:56.999645 17754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:57.000526 17754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:57.001745 17754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:57.001822 17754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:57.001947 17754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:57.001997 17754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:57.031247 17754 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:34613
I20260924 10:01:57.031296 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:34613 every 8 connection(s)
I20260924 10:01:57.032491 17754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:01:57.033957 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17754
I20260924 10:01:57.034142 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:01:57.034677 13960 external_mini_cluster.cc:1428] Setting key 772985183349a510dfa672681c485453
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244116, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:01:57.052173 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.035228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:51931 (local address 127.13.162.62:42217)
0924 10:01:57.035430 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.035434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.036661 (+ 1227us) server_negotiation.cc:407] Connection header received
0924 10:01:57.038539 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.038544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.038621 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.038701 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.040159 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.040856 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.042397 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.042810 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.042915 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.046384 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.046391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.046396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.046428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.049223 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.049930 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.049936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.049940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.050059 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.050468 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.050473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.050476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.050733 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.050852 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.051607 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.051833 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:01:57.053223 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.035772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42217 (local address 127.13.162.2:51931)
0924 10:01:57.036515 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:57.036564 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:57.038239 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:57.038945 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:57.038953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:57.039446 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:57.040003 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:57.040021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.040997 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.041002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:57.042225 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:57.042239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.043010 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.043017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:57.043225 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.043816 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:57.043900 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:57.046131 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:57.049352 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:57.049361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:57.049380 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:57.049817 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:57.050157 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:57.050162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:57.050166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:57.050376 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:57.050853 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:57.050861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:57.051212 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 10:01:57.051489 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 10:01:57.052028 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260924 10:01:57.054687 17882 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42217
I20260924 10:01:57.055011 17882 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:57.055790 17882 heartbeater.cc:511] Master 127.13.162.62:42217 requested a full tablet report, sending...
I20260924 10:01:57.057397 17559 ts_manager.cc:194] Registered new tserver with Master: 1488b112762e45998da1623755130c29 (127.13.162.2:34613)
I20260924 10:01:57.058482 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:51931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:01:57.073352 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42217
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:57.373541 17887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:57.373826 17887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:57.373946 17887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:57.381038 17887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:57.381174 17887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:57.381323 17887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:01:57.391013 17887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:57.392261 17887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:57.393548 17887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:57.405647 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:57.405727 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:57.405730 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:57.406548 17887 server_base.cc:1034] running on GCE node
I20260924 10:01:57.407110 17887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:57.407970 17887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:57.409155 17887 hybrid_clock.cc:648] HybridClock initialized: now 1790244117409111 us; error 37 us; skew 500 ppm
I20260924 10:01:57.409447 17887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:57.412281 17887 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:01:57.413671 17887 webserver.cc:460] Webserver started at http://127.13.162.3:42779/ using document root <none> and password file <none>
I20260924 10:01:57.414338 17887 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:57.414469 17887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:57.414752 17887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:57.417515 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "7c36f40477414d699726f29c80f4b3d8"
format_stamp: "Formatted at 2026-09-24 10:01:57 on dist-test-slave-jvsj"
server_key: "ac2601d5392b817336d64378b8389140"
server_key_iv: "7164db5c1fafa8cc5039ab8340d92aec"
server_key_version: "encryptionkey@0"
I20260924 10:01:57.418243 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "7c36f40477414d699726f29c80f4b3d8"
format_stamp: "Formatted at 2026-09-24 10:01:57 on dist-test-slave-jvsj"
server_key: "ac2601d5392b817336d64378b8389140"
server_key_iv: "7164db5c1fafa8cc5039ab8340d92aec"
server_key_version: "encryptionkey@0"
I20260924 10:01:57.423209 17887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:01:57.426594 17902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:57.428030 17887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:01:57.428169 17887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "7c36f40477414d699726f29c80f4b3d8"
format_stamp: "Formatted at 2026-09-24 10:01:57 on dist-test-slave-jvsj"
server_key: "ac2601d5392b817336d64378b8389140"
server_key_iv: "7164db5c1fafa8cc5039ab8340d92aec"
server_key_version: "encryptionkey@0"
I20260924 10:01:57.428345 17887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:57.445328 17887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:57.449236 17887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:57.449501 17887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:57.450455 17887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:01:57.451774 17887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:01:57.451845 17887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:57.451947 17887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:01:57.451989 17887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:57.480736 17887 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:45557
I20260924 10:01:57.480769 18014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:45557 every 8 connection(s)
I20260924 10:01:57.482010 17887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:01:57.488014 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 17887
I20260924 10:01:57.488201 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:01:57.488582 13960 external_mini_cluster.cc:1428] Setting key 860c2bff1301ab591cfc69529212bb6a
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:57.500967 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.484513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:55153 (local address 127.13.162.62:42217)
0924 10:01:57.484700 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.484706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.485889 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 10:01:57.487714 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.487720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.487806 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.487934 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.489586 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.490555 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.492122 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.492509 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.492598 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.495769 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.495774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.495777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.495808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.498125 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.498801 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.498805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.498808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.499070 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.499507 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.499510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.499512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.499771 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.499872 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.500525 (+ 653us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.500678 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:01:57.501871 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.485038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42217 (local address 127.13.162.3:55153)
0924 10:01:57.485710 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:57.485775 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:57.487431 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:57.488140 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:57.488152 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:57.488731 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:57.489420 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:57.489445 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.490692 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.490696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:57.491997 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:57.492009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.492673 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.492677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:57.492952 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.493525 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:57.493546 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:57.495574 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:57.498232 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:57.498238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:57.498251 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:57.498696 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:57.499198 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:57.499202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:57.499204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:57.499410 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:57.499876 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:57.499881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:57.500184 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 10:01:57.500431 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 10:01:57.500799 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":256,"threads_started":1}
I20260924 10:01:57.503144 18015 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42217
I20260924 10:01:57.503463 18015 heartbeater.cc:464] Registering TS with master...
I20260924 10:01:57.504153 18015 heartbeater.cc:511] Master 127.13.162.62:42217 requested a full tablet report, sending...
I20260924 10:01:57.505409 17559 ts_manager.cc:194] Registered new tserver with Master: 7c36f40477414d699726f29c80f4b3d8 (127.13.162.3:45557)
I20260924 10:01:57.506156 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:55153
I20260924 10:01:57.516074 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:01:57.560590 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.546175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.13.162.1:35795)
0924 10:01:57.546711 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.546718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.546746 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:01:57.546931 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.546939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.547159 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.547287 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.548559 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.549514 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.550686 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.551228 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.551321 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.554991 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.555002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.555017 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.555057 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.557770 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.558554 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.558561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.558563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.558695 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.559156 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.559160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.559163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.559524 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.559757 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.560034 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.560229 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":285,"threads_started":1}
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:57.577548 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.564555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.13.162.62:42217)
0924 10:01:57.564758 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.564764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.564933 (+ 169us) server_negotiation.cc:407] Connection header received
0924 10:01:57.565060 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.565064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.565124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.565196 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.566241 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.567242 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.568436 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.569040 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.569166 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.573016 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.573021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.573024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.573056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.575501 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.576042 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.576045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.576047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.576172 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.576429 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.576432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.576434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.576743 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.576923 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.577174 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.577325 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:01:57.578088 17559 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662
I20260924 10:01:57.578935 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17620
I20260924 10:01:57.590091 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.579709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.13.162.1:35795)
0924 10:01:57.579946 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.579951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.580125 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:01:57.580250 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.580253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.580337 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.580412 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.581433 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.582111 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.583319 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.583700 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.583791 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.585728 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.585733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.585736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.585766 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.588140 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.588671 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.588674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.588677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.588780 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.589029 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.589032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.589034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.589249 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.589414 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.589663 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.589825 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:01:57.590684 17743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:57.670208 17748 heartbeater.cc:503] Master 127.13.162.62:42217 was elected leader, sending a full tablet report...
I20260924 10:01:57.670737 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17620
W20260924 10:01:57.683698 17544 connection.cc:535] server connection from 127.13.162.1:46363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:01:57.683959 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17754
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:01:57.699055 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.684837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.13.162.2:34613)
0924 10:01:57.685354 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.685361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.685387 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:01:57.685444 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.685451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.685662 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.685785 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.686948 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.688029 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.689231 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.689776 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.689890 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.693702 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.693714 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.693730 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.693765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.696551 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.697153 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.697160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.697162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.697273 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.697620 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.697624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.697626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.697978 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.698204 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.698471 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.698664 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260924 10:01:57.700254 17877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:57.781176 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17754
I20260924 10:01:57.793663 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17887
Sep 24 10:01:57 dist-test-slave-jvsj krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:01:57.809052 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:57.794668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.13.162.3:45557)
0924 10:01:57.795264 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:57.795271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:57.795299 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:01:57.795359 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:57.795366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:57.795577 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:57.795726 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:57.796998 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.798013 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.799293 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:57.799843 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:57.799960 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:57.803956 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:57.803972 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:57.803990 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:57.804026 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:57.806517 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.807113 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.807120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.807122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.807236 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:57.807586 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:57.807589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:57.807591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:57.807923 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:57.808194 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:57.808478 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 10:01:57.808724 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
I20260924 10:01:57.810451 18010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:57.889643 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17887
I20260924 10:01:57.901490 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 17527
I20260924 10:01:57.902751 17589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:58.001088 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 17527
2026-09-24T10:01:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3591 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:58 dist-test-slave-jvsj krb5kdc[18034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:58 dist-test-slave-jvsj krb5kdc[18034](info): set up 2 sockets
Sep 24 10:01:58 dist-test-slave-jvsj krb5kdc[18034](info): commencing operation
krb5kdc: starting...
W20260924 10:01:58.994423 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:01:59 dist-test-slave-jvsj krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:01:59.163830 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:43717
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:43717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:59.467500 18047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:59.467770 18047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:59.467905 18047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:59.474957 18047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:59.475065 18047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:59.475121 18047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:59.475194 18047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:59.475247 18047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:59.485343 18047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:43717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:43717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:59.486660 18047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:59.487977 18047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:59.498821 18053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:59.498849 18055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:59.499107 18052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:59.499603 18047 server_base.cc:1034] running on GCE node
I20260924 10:01:59.500324 18047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:59.501693 18047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:59.502882 18047 hybrid_clock.cc:648] HybridClock initialized: now 1790244119502809 us; error 64 us; skew 500 ppm
I20260924 10:01:59.503196 18047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:59 dist-test-slave-jvsj krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244119, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:59.506047 18047 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:01:59.507194 18047 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:59.507309 18047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:59.507640 18047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:59.510545 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "0451f4232bed485ebe861506b3039247"
format_stamp: "Formatted at 2026-09-24 10:01:59 on dist-test-slave-jvsj"
server_key: "853e62f20c3dfeece0523de9721709ba"
server_key_iv: "239a48e15d8e3b94981076cfa43e1542"
server_key_version: "encryptionkey@0"
I20260924 10:01:59.511298 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "0451f4232bed485ebe861506b3039247"
format_stamp: "Formatted at 2026-09-24 10:01:59 on dist-test-slave-jvsj"
server_key: "853e62f20c3dfeece0523de9721709ba"
server_key_iv: "239a48e15d8e3b94981076cfa43e1542"
server_key_version: "encryptionkey@0"
I20260924 10:01:59.516088 18047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:01:59.519424 18061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:59.520762 18047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:59.520917 18047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "0451f4232bed485ebe861506b3039247"
format_stamp: "Formatted at 2026-09-24 10:01:59 on dist-test-slave-jvsj"
server_key: "853e62f20c3dfeece0523de9721709ba"
server_key_iv: "239a48e15d8e3b94981076cfa43e1542"
server_key_version: "encryptionkey@0"
I20260924 10:01:59.521106 18047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:59.537067 18047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:59.541097 18047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:59.541334 18047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:59.557539 18047 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:43717
I20260924 10:01:59.557561 18112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:43717 every 8 connection(s)
I20260924 10:01:59.558455 18047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:59.559026 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18047
I20260924 10:01:59.559237 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:01:59.559942 13960 external_mini_cluster.cc:1428] Setting key af1448d82617d4c6ca7817c3583d2390
I20260924 10:01:59.562693 18113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:01:59 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:01:59.572775 18113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247: Bootstrap starting.
I20260924 10:01:59.576076 18113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:59.577085 18113 log.cc:824] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:59.580037 18113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247: No bootstrap required, opened a new log
I20260924 10:01:59.580510 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:59.561898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.13.162.62:43717)
0924 10:01:59.562685 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:59.562697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:59.562737 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:01:59.564346 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:59.564369 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:59.564727 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:59.565100 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:59.566510 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:59.567799 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:59.568979 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:59.569652 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:59.569848 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:59.573387 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:59.573394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:59.573408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:59.573445 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:59.576457 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:59.577090 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:59.577096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:59.577102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:59.577278 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:59.577674 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:59.577678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:59.577682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:59.578219 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:59.578520 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:59.578859 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:01:59.579258 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":594,"thread_start_us":399,"threads_started":1}
I20260924 10:01:59.587791 18113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0451f4232bed485ebe861506b3039247" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43717 } }
I20260924 10:01:59.588116 18113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:59.588187 18113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0451f4232bed485ebe861506b3039247, State: Initialized, Role: FOLLOWER
I20260924 10:01:59.588637 18113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0451f4232bed485ebe861506b3039247" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43717 } }
I20260924 10:01:59.588827 18113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:59.588927 18113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:59.589046 18113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:59.591243 18113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0451f4232bed485ebe861506b3039247" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43717 } }
I20260924 10:01:59.591663 18113 leader_election.cc:302] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0451f4232bed485ebe861506b3039247; no voters:
I20260924 10:01:59.592149 18113 leader_election.cc:288] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:59.592272 18118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:59.592974 18118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [term 1 LEADER]: Becoming Leader. State: Replica: 0451f4232bed485ebe861506b3039247, State: Running, Role: LEADER
I20260924 10:01:59.593458 18118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0451f4232bed485ebe861506b3039247" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43717 } }
I20260924 10:01:59.594076 18113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:59.596314 18120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0451f4232bed485ebe861506b3039247. Latest consensus state: current_term: 1 leader_uuid: "0451f4232bed485ebe861506b3039247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0451f4232bed485ebe861506b3039247" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43717 } } }
I20260924 10:01:59.596315 18119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0451f4232bed485ebe861506b3039247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0451f4232bed485ebe861506b3039247" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43717 } } }
I20260924 10:01:59.596701 18119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:59.596705 18120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:59.598639 18125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:59.602878 18125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:59.610478 18125 catalog_manager.cc:1347] Generated new cluster ID: 4287594326674261b030f75de3a4734e
I20260924 10:01:59.610580 18125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:59.624270 18125 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:59.626065 18125 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:59.634758 18125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247: Generated new TSK 0
I20260924 10:01:59.635771 18125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:01:59.694130 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43717
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:00.018812 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:00.019135 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:00.019275 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:00.026666 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:00.026803 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:00.026966 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:00.036814 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:00.038157 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:00.039482 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:00.051864 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:00.051939 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:00.052153 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:00.052536 18139 server_base.cc:1034] running on GCE node
I20260924 10:02:00.053156 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:00.054082 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:00.055269 18139 hybrid_clock.cc:648] HybridClock initialized: now 1790244120055210 us; error 52 us; skew 500 ppm
I20260924 10:02:00.055589 18139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:00 dist-test-slave-jvsj krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:00.058707 18139 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:02:00.059820 18139 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:00.059911 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:00.060204 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:00.063197 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "a13db48c86154c4f9f2585e87eebe26b"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "e37bd0568ecd345d6b8f47e30243bab1"
server_key_iv: "789e9385b374c54588ba6bf0a445d134"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.063983 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "a13db48c86154c4f9f2585e87eebe26b"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "e37bd0568ecd345d6b8f47e30243bab1"
server_key_iv: "789e9385b374c54588ba6bf0a445d134"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.069056 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:00.072548 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:00.073963 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:00.074157 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "a13db48c86154c4f9f2585e87eebe26b"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "e37bd0568ecd345d6b8f47e30243bab1"
server_key_iv: "789e9385b374c54588ba6bf0a445d134"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.074406 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:00.091863 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:00.096236 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:00.096480 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:00.097497 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:00.098932 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:00.099033 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:00.099133 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:00.099205 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:00.128249 18139 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41313
I20260924 10:02:00.128298 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41313 every 8 connection(s)
I20260924 10:02:00.129182 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:00.134087 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18139
I20260924 10:02:00.134275 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:00.134948 13960 external_mini_cluster.cc:1428] Setting key c951fa7ca4e71e7741a56dc92869909b
Sep 24 10:02:00 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:00.149065 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:00.131568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:34943 (local address 127.13.162.62:43717)
0924 10:02:00.132024 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:00.132028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:00.133053 (+ 1025us) server_negotiation.cc:407] Connection header received
0924 10:02:00.135085 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:00.135092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:00.135174 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:00.135286 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:00.137028 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.138094 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.139656 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.140197 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.140314 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:00.143744 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:00.143749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:00.143752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:00.143787 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:00.146133 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:00.146889 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:00.146893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:00.146896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:00.147007 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:00.147400 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:00.147403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:00.147405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:00.147665 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:00.147790 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:00.148486 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 10:02:00.148698 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":214,"threads_started":1}
I20260924 10:02:00.150050 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:00.132151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43717 (local address 127.13.162.1:34943)
0924 10:02:00.132844 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:00.132882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:00.134755 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:00.135517 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:00.135525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:00.136170 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:00.136828 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:00.136869 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.138236 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.138240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:00.139527 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:00.139539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.140383 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.140389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:00.140585 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:00.141237 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:00.141259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:00.143524 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:00.146266 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:00.146276 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:00.146294 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:00.146766 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:00.147108 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:00.147113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:00.147117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:00.147287 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:00.147780 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:00.147788 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:00.148123 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 10:02:00.148348 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 10:02:00.148892 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:02:00.151701 18266 heartbeater.cc:347] Connected to a master server at 127.13.162.62:43717
I20260924 10:02:00.152040 18266 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:00.152694 18266 heartbeater.cc:511] Master 127.13.162.62:43717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:02:00.154737 18078 ts_manager.cc:194] Registered new tserver with Master: a13db48c86154c4f9f2585e87eebe26b (127.13.162.1:41313)
I20260924 10:02:00.156105 18078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:34943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:02:00.173632 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43717
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:00.489534 18272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:00.489838 18272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:00.489995 18272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:00.497372 18272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:00.497491 18272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:00.497639 18272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:00.507616 18272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:00.508983 18272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:00.510344 18272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:00.522771 18277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:00.522882 18280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:00.522725 18278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:00.523633 18272 server_base.cc:1034] running on GCE node
I20260924 10:02:00.524253 18272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:00.525271 18272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:00.526438 18272 hybrid_clock.cc:648] HybridClock initialized: now 1790244120526359 us; error 62 us; skew 500 ppm
I20260924 10:02:00.526758 18272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:00 dist-test-slave-jvsj krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:00.529870 18272 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:02:00.531030 18272 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:00.531131 18272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:00.531431 18272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:00.534440 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "574cc3a42ec8421fbfc45b1dd8e2007e"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "5b780feccb56e11ab1e5ff2ba68172ba"
server_key_iv: "ee963fa97253b1dc25b72225461d9c8b"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.535236 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "574cc3a42ec8421fbfc45b1dd8e2007e"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "5b780feccb56e11ab1e5ff2ba68172ba"
server_key_iv: "ee963fa97253b1dc25b72225461d9c8b"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.540271 18272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:02:00.544085 18286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:00.545562 18272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:02:00.545717 18272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "574cc3a42ec8421fbfc45b1dd8e2007e"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "5b780feccb56e11ab1e5ff2ba68172ba"
server_key_iv: "ee963fa97253b1dc25b72225461d9c8b"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.545929 18272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:00.557065 18272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:00.562389 18272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:00.562682 18272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:00.563830 18272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:00.565464 18272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:00.565578 18272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:00.565678 18272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:00.565749 18272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:00.595963 18272 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:38089
I20260924 10:02:00.596035 18398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:38089 every 8 connection(s)
I20260924 10:02:00.596773 18272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:00.601056 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18272
I20260924 10:02:00.601250 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:00.601790 13960 external_mini_cluster.cc:1428] Setting key 715225c6e17ccb309bcfd5018cab5890
Sep 24 10:02:00 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:00.617326 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:00.599138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:60639 (local address 127.13.162.62:43717)
0924 10:02:00.599335 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:00.599340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:00.600572 (+ 1232us) server_negotiation.cc:407] Connection header received
0924 10:02:00.602845 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:00.602851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:00.602935 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:00.603032 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:00.604824 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.605534 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.607572 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.607972 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.608076 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:00.611753 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:00.611758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:00.611762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:00.611793 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:00.614097 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:00.614944 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:00.614947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:00.614950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:00.615099 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:00.615560 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:00.615564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:00.615566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:00.615858 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:00.615972 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:00.616889 (+ 917us) server_negotiation.cc:299] Negotiation successful
0924 10:02:00.617082 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:02:00.618909 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:00.599741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43717 (local address 127.13.162.2:60639)
0924 10:02:00.600421 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:00.600458 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:00.602593 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:00.603271 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:00.603280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:00.603842 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:00.604655 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:00.604679 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.605681 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.605687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:00.607426 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:00.607440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:00.608175 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:00.608180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:00.608435 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:00.609152 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:00.609181 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:00.611530 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:00.614223 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:00.614232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:00.614248 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:00.614843 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:00.615194 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:00.615198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:00.615202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:00.615450 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:00.615983 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:00.615990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:00.616465 (+ 475us) client_negotiation.cc:769] Sending connection context
0924 10:02:00.616790 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 10:02:00.617346 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:02:00.620273 18399 heartbeater.cc:347] Connected to a master server at 127.13.162.62:43717
I20260924 10:02:00.620591 18399 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:00.621331 18399 heartbeater.cc:511] Master 127.13.162.62:43717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:02:00.622794 18078 ts_manager.cc:194] Registered new tserver with Master: 574cc3a42ec8421fbfc45b1dd8e2007e (127.13.162.2:38089)
I20260924 10:02:00.623586 18078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:60639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:02:00.643008 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43717
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:00.952260 18404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:00.952538 18404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:00.952663 18404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:00.959782 18404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:00.959913 18404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:00.960070 18404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:00.969554 18404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:00.970814 18404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:00.972079 18404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:00.984032 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:00.984062 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:00.984120 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:00.984735 18404 server_base.cc:1034] running on GCE node
I20260924 10:02:00.985252 18404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:00.986097 18404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:00.987296 18404 hybrid_clock.cc:648] HybridClock initialized: now 1790244120987230 us; error 55 us; skew 500 ppm
I20260924 10:02:00.987600 18404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:00 dist-test-slave-jvsj krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:00.994369 18404 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:02:00.995555 18404 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:00.995657 18404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:00.995960 18404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:00.998936 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "890fae10aa08ee231029a1ba06a694ec"
server_key_iv: "bf49552ee5b6640f389aa6be1d24800b"
server_key_version: "encryptionkey@0"
I20260924 10:02:00.999732 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "890fae10aa08ee231029a1ba06a694ec"
server_key_iv: "bf49552ee5b6640f389aa6be1d24800b"
server_key_version: "encryptionkey@0"
I20260924 10:02:01.004534 18404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:02:01.007905 18418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:01.009254 18404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:02:01.009415 18404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee"
format_stamp: "Formatted at 2026-09-24 10:02:00 on dist-test-slave-jvsj"
server_key: "890fae10aa08ee231029a1ba06a694ec"
server_key_iv: "bf49552ee5b6640f389aa6be1d24800b"
server_key_version: "encryptionkey@0"
I20260924 10:02:01.009604 18404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:01.023377 18404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:01.027489 18404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:01.027729 18404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:01.028628 18404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:01.029891 18404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:01.029980 18404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:01.030086 18404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:01.030156 18404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:01.058933 18404 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:44093
I20260924 10:02:01.058983 18530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:44093 every 8 connection(s)
I20260924 10:02:01.059803 18404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:01.069607 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18404
I20260924 10:02:01.069782 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:01.070188 13960 external_mini_cluster.cc:1428] Setting key a325843a8022c4093a038b902c8cbec6
Sep 24 10:02:01 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:01.078964 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.062201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:45161 (local address 127.13.162.62:43717)
0924 10:02:01.062436 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.062441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.063623 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 10:02:01.065502 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.065508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.065588 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.065675 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:01.067257 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.067968 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.069490 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.069946 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.070059 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.073745 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:01.073751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:01.073756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:01.073789 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:01.076018 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.076699 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.076703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.076706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.076818 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.077206 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.077210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.077212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.077486 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:01.077620 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.078391 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.078598 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:02:01.079880 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.062745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43717 (local address 127.13.162.3:45161)
0924 10:02:01.063453 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:01.063496 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:01.065212 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:01.065932 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:01.065941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:01.066512 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:01.067096 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.067112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.068108 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.068112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.069353 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.069366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.070170 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.070176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.070440 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.071069 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:01.071096 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:01.073547 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:01.076140 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:01.076148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:01.076165 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:01.076581 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:01.076926 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:01.076930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:01.076934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:01.077101 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:01.077611 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:01.077619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:01.078022 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 10:02:01.078256 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 10:02:01.078784 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":284,"threads_started":1}
I20260924 10:02:01.081205 18531 heartbeater.cc:347] Connected to a master server at 127.13.162.62:43717
I20260924 10:02:01.081482 18531 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:01.082120 18531 heartbeater.cc:511] Master 127.13.162.62:43717 requested a full tablet report, sending...
I20260924 10:02:01.083492 18078 ts_manager.cc:194] Registered new tserver with Master: 8445cbd9eaca4aba8c7f1b7b7e29f5ee (127.13.162.3:44093)
I20260924 10:02:01.084347 18078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:45161
I20260924 10:02:01.085367 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:02:01.100934 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.089542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.13.162.62:43717)
0924 10:02:01.089695 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.089699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.090097 (+ 398us) server_negotiation.cc:407] Connection header received
0924 10:02:01.090238 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.090241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.090295 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.090417 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:01.091745 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.092427 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.093589 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.094087 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.094184 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.096339 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:01.096343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:01.096346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:01.096376 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:01.098708 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.099324 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.099329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.099331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.099508 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.099827 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.099830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.099832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.100128 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:01.100312 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.100541 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.100697 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:02:01.104761 18078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:01.111348 18078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:01.144858 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.122727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41313 (local address 127.0.0.1:41324)
0924 10:02:01.135594 (+ 12867us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:01.135638 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:01.135851 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:01.137185 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:01.137193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:01.137226 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:01.137666 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.137682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.139991 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.139998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.142094 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.142106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.143819 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.143825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.144052 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.144142 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 10:02:01.144261 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:02:01.144455 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12630,"mutex_wait_us":3,"thread_start_us":13447,"threads_started":1}
I20260924 10:02:01.145287 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.123254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.13.162.1:41313)
0924 10:02:01.134866 (+ 11612us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.134875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.136046 (+ 1171us) server_negotiation.cc:407] Connection header received
0924 10:02:01.136168 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.136176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.136448 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.137920 (+ 1472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:01.138156 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.139823 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.142251 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.143569 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.144534 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.144767 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.144847 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.145001 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11398,"thread_start_us":11211,"threads_started":1}
I20260924 10:02:01.146430 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.122830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.13.162.3:44093)
0924 10:02:01.135146 (+ 12316us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.135154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.136640 (+ 1486us) server_negotiation.cc:407] Connection header received
0924 10:02:01.136748 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.136756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.136950 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.137121 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:01.137983 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.141173 (+ 3190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.143710 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.145138 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.145260 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.145474 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.145954 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.146108 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12150,"thread_start_us":11989,"threads_started":1}
I20260924 10:02:01.146445 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.122755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:44093 (local address 127.0.0.1:39120)
0924 10:02:01.136457 (+ 13702us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:01.136477 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:01.136594 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:01.137393 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:01.137398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:01.137437 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:01.137792 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.137800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.141347 (+ 3547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.141352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.143532 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.143546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.145602 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.145609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.145774 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.145797 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:02:01.145876 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:02:01.145998 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13591,"thread_start_us":12460,"threads_started":1}
I20260924 10:02:01.149652 18201 tablet_service.cc:1467] Processing CreateTablet for tablet 1744ece062db42c58a0115aa5277a8f3 (DEFAULT_TABLE table=test-table [id=70d8b545bd5c4de086921c404a8c33a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:01.149796 18466 tablet_service.cc:1467] Processing CreateTablet for tablet 1744ece062db42c58a0115aa5277a8f3 (DEFAULT_TABLE table=test-table [id=70d8b545bd5c4de086921c404a8c33a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:01.150806 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.136404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.13.162.2:38089)
0924 10:02:01.141322 (+ 4918us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.141331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.142792 (+ 1461us) server_negotiation.cc:407] Connection header received
0924 10:02:01.142946 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.142954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.143220 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.143415 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:01.144366 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.145912 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.148359 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.149733 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.149877 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.150053 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.150288 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.150487 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4755,"thread_start_us":4620,"threads_started":1}
I20260924 10:02:01.150868 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.138032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:38089 (local address 127.0.0.1:51430)
0924 10:02:01.142664 (+ 4632us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:01.142681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:01.142804 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:01.143660 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:01.143665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:01.143691 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:01.144173 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.144186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.146062 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.146067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.148181 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.148196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.149945 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.149951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.150110 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.150141 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:02:01.150236 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:02:01.150498 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4493,"mutex_wait_us":24,"thread_start_us":4337,"threads_started":1}
I20260924 10:02:01.151194 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1744ece062db42c58a0115aa5277a8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:01.151196 18466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1744ece062db42c58a0115aa5277a8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:01.154309 18334 tablet_service.cc:1467] Processing CreateTablet for tablet 1744ece062db42c58a0115aa5277a8f3 (DEFAULT_TABLE table=test-table [id=70d8b545bd5c4de086921c404a8c33a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:01.155705 18334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1744ece062db42c58a0115aa5277a8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:01.158900 18266 heartbeater.cc:503] Master 127.13.162.62:43717 was elected leader, sending a full tablet report...
I20260924 10:02:01.163669 18549 tablet_bootstrap.cc:492] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Bootstrap starting.
I20260924 10:02:01.165721 18550 tablet_bootstrap.cc:492] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Bootstrap starting.
I20260924 10:02:01.167742 18549 tablet_bootstrap.cc:654] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:01.168761 18549 log.cc:824] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:01.169333 18550 tablet_bootstrap.cc:654] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:01.170578 18550 log.cc:824] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:01.171625 18549 tablet_bootstrap.cc:492] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: No bootstrap required, opened a new log
I20260924 10:02:01.171940 18549 ts_tablet_manager.cc:1397] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:02:01.172482 18553 tablet_bootstrap.cc:492] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Bootstrap starting.
I20260924 10:02:01.174286 18550 tablet_bootstrap.cc:492] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: No bootstrap required, opened a new log
I20260924 10:02:01.174587 18550 ts_tablet_manager.cc:1397] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260924 10:02:01.175906 18553 tablet_bootstrap.cc:654] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:01.177100 18553 log.cc:824] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:01.180160 18553 tablet_bootstrap.cc:492] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: No bootstrap required, opened a new log
I20260924 10:02:01.180426 18553 ts_tablet_manager.cc:1397] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:02:01.180738 18549 raft_consensus.cc:348] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.181176 18549 raft_consensus.cc:374] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:01.181279 18549 raft_consensus.cc:729] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8445cbd9eaca4aba8c7f1b7b7e29f5ee, State: Initialized, Role: FOLLOWER
I20260924 10:02:01.181886 18549 consensus_queue.cc:260] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.182981 18531 heartbeater.cc:503] Master 127.13.162.62:43717 was elected leader, sending a full tablet report...
I20260924 10:02:01.182919 18550 raft_consensus.cc:348] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.183267 18550 raft_consensus.cc:374] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:01.183357 18550 raft_consensus.cc:729] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13db48c86154c4f9f2585e87eebe26b, State: Initialized, Role: FOLLOWER
I20260924 10:02:01.183909 18550 consensus_queue.cc:260] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.184846 18549 ts_tablet_manager.cc:1428] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 10:02:01.186440 18550 ts_tablet_manager.cc:1428] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260924 10:02:01.190719 18553 raft_consensus.cc:348] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.191044 18553 raft_consensus.cc:374] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:01.191126 18553 raft_consensus.cc:729] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574cc3a42ec8421fbfc45b1dd8e2007e, State: Initialized, Role: FOLLOWER
I20260924 10:02:01.191601 18553 consensus_queue.cc:260] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.192478 18399 heartbeater.cc:503] Master 127.13.162.62:43717 was elected leader, sending a full tablet report...
I20260924 10:02:01.192873 18553 ts_tablet_manager.cc:1428] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 10:02:01.225664 18555 raft_consensus.cc:482] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:01.225973 18555 raft_consensus.cc:504] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.227631 18555 leader_election.cc:288] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a13db48c86154c4f9f2585e87eebe26b (127.13.162.1:41313), 574cc3a42ec8421fbfc45b1dd8e2007e (127.13.162.2:38089)
I20260924 10:02:01.233528 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.227887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:38089 (local address 127.13.162.3:57281)
0924 10:02:01.228266 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:01.228289 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:01.228435 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:01.228788 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:01.228792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:01.228813 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:01.229029 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.229042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.230150 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.230155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.231574 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.231585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.232856 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.232862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.233052 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.233084 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:02:01.233165 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:02:01.233304 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260924 10:02:01.233579 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.228078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:57281 (local address 127.13.162.2:38089)
0924 10:02:01.228271 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.228277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.228376 (+ 99us) server_negotiation.cc:407] Connection header received
0924 10:02:01.228586 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.228592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.228672 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.228750 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:01.229166 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.230019 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.231745 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.232712 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.232826 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.232876 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.233204 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.233332 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:02:01.234267 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.228124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46645 (local address 127.13.162.1:41313)
0924 10:02:01.228294 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.228300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.228730 (+ 430us) server_negotiation.cc:407] Connection header received
0924 10:02:01.228897 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.228901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.228982 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.229063 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:01.229598 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.230408 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.232227 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.233077 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.233220 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.233283 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.233920 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.234049 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:02:01.234763 18354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1744ece062db42c58a0115aa5277a8f3" candidate_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" is_pre_election: true
I20260924 10:02:01.234997 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.227883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41313 (local address 127.13.162.3:46645)
0924 10:02:01.228619 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:01.228640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:01.228769 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:01.229097 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:01.229102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:01.229127 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:01.229447 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.229459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.230573 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.230578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.232082 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:01.232095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.233269 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.233275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:01.233450 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.233483 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:02:01.234531 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0924 10:02:01.234673 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":51,"thread_start_us":364,"threads_started":1}
I20260924 10:02:01.235307 18354 raft_consensus.cc:2454] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8445cbd9eaca4aba8c7f1b7b7e29f5ee in term 0.
I20260924 10:02:01.235936 18420 leader_election.cc:302] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574cc3a42ec8421fbfc45b1dd8e2007e, 8445cbd9eaca4aba8c7f1b7b7e29f5ee; no voters:
I20260924 10:02:01.235935 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1744ece062db42c58a0115aa5277a8f3" candidate_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13db48c86154c4f9f2585e87eebe26b" is_pre_election: true
I20260924 10:02:01.236274 18555 raft_consensus.cc:2781] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:01.236389 18555 raft_consensus.cc:482] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:01.236456 18555 raft_consensus.cc:3037] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:01.236415 18221 raft_consensus.cc:2454] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8445cbd9eaca4aba8c7f1b7b7e29f5ee in term 0.
I20260924 10:02:01.238587 18555 raft_consensus.cc:504] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.239140 18555 leader_election.cc:288] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [CANDIDATE]: Term 1 election: Requested vote from peers a13db48c86154c4f9f2585e87eebe26b (127.13.162.1:41313), 574cc3a42ec8421fbfc45b1dd8e2007e (127.13.162.2:38089)
I20260924 10:02:01.239554 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1744ece062db42c58a0115aa5277a8f3" candidate_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13db48c86154c4f9f2585e87eebe26b"
I20260924 10:02:01.239588 18354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1744ece062db42c58a0115aa5277a8f3" candidate_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e"
I20260924 10:02:01.239724 18221 raft_consensus.cc:3037] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:01.239746 18354 raft_consensus.cc:3037] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:01.241822 18221 raft_consensus.cc:2454] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8445cbd9eaca4aba8c7f1b7b7e29f5ee in term 1.
I20260924 10:02:01.241823 18354 raft_consensus.cc:2454] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8445cbd9eaca4aba8c7f1b7b7e29f5ee in term 1.
I20260924 10:02:01.242170 18420 leader_election.cc:302] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574cc3a42ec8421fbfc45b1dd8e2007e, 8445cbd9eaca4aba8c7f1b7b7e29f5ee; no voters:
I20260924 10:02:01.242440 18555 raft_consensus.cc:2781] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:01.242883 18555 raft_consensus.cc:686] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 1 LEADER]: Becoming Leader. State: Replica: 8445cbd9eaca4aba8c7f1b7b7e29f5ee, State: Running, Role: LEADER
I20260924 10:02:01.243415 18555 consensus_queue.cc:237] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } }
I20260924 10:02:01.247677 18078 catalog_manager.cc:5495] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 8445cbd9eaca4aba8c7f1b7b7e29f5ee (127.13.162.3). New cstate: current_term: 1 leader_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8445cbd9eaca4aba8c7f1b7b7e29f5ee" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 44093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20260924 10:02:01.270653 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.265529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.13.162.3:44093)
0924 10:02:01.265731 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.265737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.265818 (+ 81us) server_negotiation.cc:407] Connection header received
0924 10:02:01.266036 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.266040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.266116 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.266248 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:01.266646 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.267355 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.268631 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.269006 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.269110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.269580 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:01.270080 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:01.270208 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.270299 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.270404 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:02:01.278939 18354 raft_consensus.cc:1261] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 1 FOLLOWER]: Refusing update from remote peer 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:01.278941 18221 raft_consensus.cc:1261] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 1 FOLLOWER]: Refusing update from remote peer 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:01.279883 18559 consensus_queue.cc:1035] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "574cc3a42ec8421fbfc45b1dd8e2007e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:01.280184 18555 consensus_queue.cc:1035] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a13db48c86154c4f9f2585e87eebe26b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:01.289583 18562 mvcc.cc:205] Tried to move back new op lower bound from 7332839920747986944 to 7332839920616710144. Current Snapshot: MvccSnapshot[applied={T|T < 7332839920747986944}]
I20260924 10:02:01.311965 18078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:01.312211 18078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:01.315645 18078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0451f4232bed485ebe861506b3039247: Sending DeleteTablet for 3 replicas of tablet 1744ece062db42c58a0115aa5277a8f3
I20260924 10:02:01.316802 18334 tablet_service.cc:1514] Processing DeleteTablet for tablet 1744ece062db42c58a0115aa5277a8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:01 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:51430
I20260924 10:02:01.316977 18466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1744ece062db42c58a0115aa5277a8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:01 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:39120
I20260924 10:02:01.317488 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet 1744ece062db42c58a0115aa5277a8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:01 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:41324
I20260924 10:02:01.318198 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18139
I20260924 10:02:01.326817 18572 tablet_replica.cc:330] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: stopping tablet replica
I20260924 10:02:01.327318 18572 raft_consensus.cc:2229] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:01.327857 18572 raft_consensus.cc:2258] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:01.329710 18573 tablet_replica.cc:330] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: stopping tablet replica
I20260924 10:02:01.330334 18574 tablet_replica.cc:330] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: stopping tablet replica
I20260924 10:02:01.330870 18574 raft_consensus.cc:2229] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:01.331353 18573 raft_consensus.cc:2229] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:01.331904 18573 raft_consensus.cc:2258] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:01.332492 18574 raft_consensus.cc:2258] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:01.334244 18574 ts_tablet_manager.cc:1905] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:01.339681 18573 ts_tablet_manager.cc:1905] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:01.339814 18572 ts_tablet_manager.cc:1905] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:01.340870 18574 ts_tablet_manager.cc:1918] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:01.341030 18574 log.cc:1196] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/1744ece062db42c58a0115aa5277a8f3
Sep 24 10:02:01 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:02:01.341511 18574 ts_tablet_manager.cc:1939] T 1744ece062db42c58a0115aa5277a8f3 P a13db48c86154c4f9f2585e87eebe26b: Deleting consensus metadata
I20260924 10:02:01.343334 18065 catalog_manager.cc:4841] TS a13db48c86154c4f9f2585e87eebe26b (127.13.162.1:41313): tablet 1744ece062db42c58a0115aa5277a8f3 (table test-table [id=70d8b545bd5c4de086921c404a8c33a7]) successfully deleted
I20260924 10:02:01.346086 18573 ts_tablet_manager.cc:1918] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:01.346227 18573 log.cc:1196] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/1744ece062db42c58a0115aa5277a8f3
I20260924 10:02:01.346304 18572 ts_tablet_manager.cc:1918] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:01.346472 18572 log.cc:1196] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/1744ece062db42c58a0115aa5277a8f3
I20260924 10:02:01.346830 18573 ts_tablet_manager.cc:1939] T 1744ece062db42c58a0115aa5277a8f3 P 8445cbd9eaca4aba8c7f1b7b7e29f5ee: Deleting consensus metadata
I20260924 10:02:01.346975 18572 ts_tablet_manager.cc:1939] T 1744ece062db42c58a0115aa5277a8f3 P 574cc3a42ec8421fbfc45b1dd8e2007e: Deleting consensus metadata
I20260924 10:02:01.348579 18063 catalog_manager.cc:4841] TS 574cc3a42ec8421fbfc45b1dd8e2007e (127.13.162.2:38089): tablet 1744ece062db42c58a0115aa5277a8f3 (table test-table [id=70d8b545bd5c4de086921c404a8c33a7]) successfully deleted
I20260924 10:02:01.348824 18063 catalog_manager.cc:4841] TS 8445cbd9eaca4aba8c7f1b7b7e29f5ee (127.13.162.3:44093): tablet 1744ece062db42c58a0115aa5277a8f3 (table test-table [id=70d8b545bd5c4de086921c404a8c33a7]) successfully deleted
I20260924 10:02:01.349260 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.320531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.13.162.1:41313)
0924 10:02:01.320708 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.320715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.330834 (+ 10119us) server_negotiation.cc:407] Connection header received
0924 10:02:01.331031 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.331037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.331134 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.331246 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:01.333191 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.334411 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.336255 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.336939 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.337040 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.342495 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:01.342509 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:01.342526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:01.342566 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:01.346030 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.346878 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.346887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.346891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.347048 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.347492 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.347496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.347499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.347928 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:01.348194 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.348726 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.348941 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:02:01.349879 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:01.435024 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18139
I20260924 10:02:01.448653 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18272
Sep 24 10:02:01 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:02:01.465020 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.449453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.13.162.2:38089)
0924 10:02:01.449600 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.449605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.449621 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:02:01.449821 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.449825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.449926 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.450015 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:01.451586 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.452371 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.454121 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.454550 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.454641 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.459300 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:01.459314 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:01.459329 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:01.459361 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:01.462220 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.463084 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.463091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.463093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.463213 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.463575 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.463579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.463581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.463922 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:01.464200 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.464497 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.464712 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:02:01.465737 18394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:01.552237 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18272
I20260924 10:02:01.565141 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18404
Sep 24 10:02:01 dist-test-slave-jvsj krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:02:01.579890 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:01.565996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.13.162.3:44093)
0924 10:02:01.566161 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:01.566167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:01.566185 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:01.566392 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:01.566396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:01.566457 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:01.566556 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:01.567759 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.568590 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.569783 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:01.570221 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:01.570488 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:01.574306 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:01.574396 (+ 90us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:01.574414 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:01.574458 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:01.577185 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.577799 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.577807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.577809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.577972 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:01.578360 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:01.578366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:01.578373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:01.578735 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:01.579006 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:01.579291 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 10:02:01.579491 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:02:01.580579 18526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:01.699321 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18404
I20260924 10:02:01.714645 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18047
I20260924 10:02:01.715749 18108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:01.823278 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18047
2026-09-24T10:02:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3822 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T10:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:01Z Disabled control of system clock
I20260924 10:02:01.882287 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:33893
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:02:02.187968 18583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:02.188271 18583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:02.188390 18583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:02.195513 18583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:02.195644 18583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:02.195708 18583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:02.195782 18583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:02.195837 18583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:02.204968 18583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:33893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:02.206234 18583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:02.207960 18583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:02.218737 18588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:02.218765 18589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:02.218840 18591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:02.219576 18583 server_base.cc:1034] running on GCE node
I20260924 10:02:02.220357 18583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:02.221740 18583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:02.223004 18583 hybrid_clock.cc:648] HybridClock initialized: now 1790244122222887 us; error 43 us; skew 500 ppm
I20260924 10:02:02.223336 18583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:02.225534 18583 webserver.cc:460] Webserver started at http://127.13.162.62:34441/ using document root <none> and password file <none>
I20260924 10:02:02.226257 18583 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:02.226382 18583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:02.226701 18583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:02.229561 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "9055a3adcad241cfb7cf02721823870a"
format_stamp: "Formatted at 2026-09-24 10:02:02 on dist-test-slave-jvsj"
server_key: "5d29ae512f1b24f9e2fd9e0da5488010"
server_key_iv: "90d8c970c35769f4a2f538b8294830ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:02.230347 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "9055a3adcad241cfb7cf02721823870a"
format_stamp: "Formatted at 2026-09-24 10:02:02 on dist-test-slave-jvsj"
server_key: "5d29ae512f1b24f9e2fd9e0da5488010"
server_key_iv: "90d8c970c35769f4a2f538b8294830ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:02.235309 18583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:02:02.238729 18597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:02.240038 18583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:02.240188 18583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "9055a3adcad241cfb7cf02721823870a"
format_stamp: "Formatted at 2026-09-24 10:02:02 on dist-test-slave-jvsj"
server_key: "5d29ae512f1b24f9e2fd9e0da5488010"
server_key_iv: "90d8c970c35769f4a2f538b8294830ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:02.240381 18583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:02.245826 18583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:02.246682 18583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:02.247023 18583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:02.263334 18583 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:33893
I20260924 10:02:02.263355 18648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:33893 every 8 connection(s)
I20260924 10:02:02.264617 18583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:02.268314 18649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:02.268492 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18583
I20260924 10:02:02.268646 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:02.269264 13960 external_mini_cluster.cc:1428] Setting key 7703847b05310ed3c8d7b4278f62aa3a
I20260924 10:02:02.277452 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:02.271161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.13.162.62:33893)
0924 10:02:02.272129 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:02.272139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:02.272178 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:02:02.274024 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:02.274046 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:02.274539 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:02.274900 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:02.275876 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:02.275883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:02.275895 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:02.275909 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:02.275970 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:02.275986 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:02.276177 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:02.276300 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 10:02:02.276546 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":310,"threads_started":1}
I20260924 10:02:02.280568 18649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a: Bootstrap starting.
I20260924 10:02:02.283494 18649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:02.284407 18649 log.cc:824] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:02.286950 18649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a: No bootstrap required, opened a new log
I20260924 10:02:02.293713 18649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055a3adcad241cfb7cf02721823870a" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33893 } }
I20260924 10:02:02.294072 18649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:02.294173 18649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055a3adcad241cfb7cf02721823870a, State: Initialized, Role: FOLLOWER
I20260924 10:02:02.294852 18649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055a3adcad241cfb7cf02721823870a" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33893 } }
I20260924 10:02:02.295076 18649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:02.295199 18649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:02.295337 18649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:02.297417 18649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055a3adcad241cfb7cf02721823870a" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33893 } }
I20260924 10:02:02.297887 18649 leader_election.cc:302] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9055a3adcad241cfb7cf02721823870a; no voters:
I20260924 10:02:02.298491 18649 leader_election.cc:288] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:02.298676 18654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:02.299338 18654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [term 1 LEADER]: Becoming Leader. State: Replica: 9055a3adcad241cfb7cf02721823870a, State: Running, Role: LEADER
I20260924 10:02:02.299849 18654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055a3adcad241cfb7cf02721823870a" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33893 } }
I20260924 10:02:02.300516 18649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:02.302834 18656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9055a3adcad241cfb7cf02721823870a. Latest consensus state: current_term: 1 leader_uuid: "9055a3adcad241cfb7cf02721823870a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055a3adcad241cfb7cf02721823870a" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33893 } } }
I20260924 10:02:02.302812 18655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9055a3adcad241cfb7cf02721823870a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055a3adcad241cfb7cf02721823870a" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33893 } } }
I20260924 10:02:02.303253 18655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:02.303251 18656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:02.304414 18660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:02.309414 18660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:02.319262 18660 catalog_manager.cc:1347] Generated new cluster ID: bea289992cda453bb7417bf1dd0f7f20
I20260924 10:02:02.319392 18660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:02.336052 18660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:02.336467 18660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:02.345674 18660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a: Generated new TSK 0
I20260924 10:02:02.346596 18660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:02:02.360762 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33893
--builtin_ntp_servers=127.13.162.20:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:02:02.677302 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:02.677561 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:02.677645 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:02.684676 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:02.684773 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:02.684877 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:02.693708 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:02.695025 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:02.696784 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:02.709385 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:02.709424 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:02.709450 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:02.710415 18673 server_base.cc:1034] running on GCE node
I20260924 10:02:02.711019 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:02.711925 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:02.713133 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790244122713086 us; error 43 us; skew 500 ppm
I20260924 10:02:02.713450 18673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:02.715695 18673 webserver.cc:460] Webserver started at http://127.13.162.1:34147/ using document root <none> and password file <none>
I20260924 10:02:02.716409 18673 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:02.716498 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:02.716817 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:02.719730 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "90d22447558b4200ba9d554d1f2236d3"
format_stamp: "Formatted at 2026-09-24 10:02:02 on dist-test-slave-jvsj"
server_key: "101f8550baa972f557bb834833cb29a4"
server_key_iv: "c0fc693ef9e4b6a524cba7e8e0fd2469"
server_key_version: "encryptionkey@0"
I20260924 10:02:02.720491 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "90d22447558b4200ba9d554d1f2236d3"
format_stamp: "Formatted at 2026-09-24 10:02:02 on dist-test-slave-jvsj"
server_key: "101f8550baa972f557bb834833cb29a4"
server_key_iv: "c0fc693ef9e4b6a524cba7e8e0fd2469"
server_key_version: "encryptionkey@0"
I20260924 10:02:02.725535 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:02.728853 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:02.730293 18673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:02.730502 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "90d22447558b4200ba9d554d1f2236d3"
format_stamp: "Formatted at 2026-09-24 10:02:02 on dist-test-slave-jvsj"
server_key: "101f8550baa972f557bb834833cb29a4"
server_key_iv: "c0fc693ef9e4b6a524cba7e8e0fd2469"
server_key_version: "encryptionkey@0"
I20260924 10:02:02.730739 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:02.736274 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:02.736987 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:02.737224 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:02.738037 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:02.739355 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:02.739445 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:02.739544 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:02.739610 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:02.767671 18673 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:38035
I20260924 10:02:02.767732 18799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:38035 every 8 connection(s)
I20260924 10:02:02.768885 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:02.776196 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:02.771289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:58893 (local address 127.13.162.62:33893)
0924 10:02:02.771454 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:02.771460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:02.772798 (+ 1338us) server_negotiation.cc:407] Connection header received
0924 10:02:02.774394 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:02.774398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:02.774448 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:02.774533 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:02.775661 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:02.775665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:02.775668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:02.775682 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:02.775712 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:02.775724 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:02.775807 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:02.775934 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 10:02:02.775977 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:02:02.777216 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:02.771801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33893 (local address 127.13.162.1:58893)
0924 10:02:02.772658 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:02.772693 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:02.774139 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:02.774718 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:02.774725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:02.774815 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:02.775400 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:02.775402 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:02.775440 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:02.775521 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:02.775791 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:02.775797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:02.775802 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:02:02.776023 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 10:02:02.776207 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":257,"threads_started":1}
I20260924 10:02:02.777364 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18673
I20260924 10:02:02.777560 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:02.778172 13960 external_mini_cluster.cc:1428] Setting key 3a35af7a908358df7d91a96219e1038e
I20260924 10:02:02.778785 18800 heartbeater.cc:347] Connected to a master server at 127.13.162.62:33893
I20260924 10:02:02.779070 18800 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:02.779615 18800 heartbeater.cc:511] Master 127.13.162.62:33893 requested a full tablet report, sending...
I20260924 10:02:02.780789 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33893
--builtin_ntp_servers=127.13.162.20:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:02:02.781698 18614 ts_manager.cc:194] Registered new tserver with Master: 90d22447558b4200ba9d554d1f2236d3 (127.13.162.1:38035)
W20260924 10:02:03.081508 18803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:03.081785 18803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:03.081892 18803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:03.088953 18803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:03.089042 18803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:03.089178 18803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:03.098078 18803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:03.099323 18803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:03.101068 18803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:03.113579 18809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:03.113664 18808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:03.113638 18811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:03.114709 18803 server_base.cc:1034] running on GCE node
I20260924 10:02:03.115312 18803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:03.116168 18803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:03.117354 18803 hybrid_clock.cc:648] HybridClock initialized: now 1790244123117311 us; error 38 us; skew 500 ppm
I20260924 10:02:03.117656 18803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:03.119779 18803 webserver.cc:460] Webserver started at http://127.13.162.2:36393/ using document root <none> and password file <none>
I20260924 10:02:03.120419 18803 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:03.120503 18803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:03.120798 18803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:03.123484 18803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "52d665453478412ba2f1944379432aee"
format_stamp: "Formatted at 2026-09-24 10:02:03 on dist-test-slave-jvsj"
server_key: "fa4eb2d48b70c0849130637a303d86a3"
server_key_iv: "704f7f7804a142e012dde99d9b56df3e"
server_key_version: "encryptionkey@0"
I20260924 10:02:03.124218 18803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "52d665453478412ba2f1944379432aee"
format_stamp: "Formatted at 2026-09-24 10:02:03 on dist-test-slave-jvsj"
server_key: "fa4eb2d48b70c0849130637a303d86a3"
server_key_iv: "704f7f7804a142e012dde99d9b56df3e"
server_key_version: "encryptionkey@0"
I20260924 10:02:03.129001 18803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:02:03.132277 18817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:03.133617 18803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:03.133754 18803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "52d665453478412ba2f1944379432aee"
format_stamp: "Formatted at 2026-09-24 10:02:03 on dist-test-slave-jvsj"
server_key: "fa4eb2d48b70c0849130637a303d86a3"
server_key_iv: "704f7f7804a142e012dde99d9b56df3e"
server_key_version: "encryptionkey@0"
I20260924 10:02:03.133962 18803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:03.139475 18803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:03.140192 18803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:03.140419 18803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:03.141244 18803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:03.142589 18803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:03.142679 18803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:03.142782 18803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:03.142846 18803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:03.171120 18803 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:45357
I20260924 10:02:03.171176 18929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:45357 every 8 connection(s)
I20260924 10:02:03.172391 18803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:03.176578 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18803
I20260924 10:02:03.176717 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:03.177134 13960 external_mini_cluster.cc:1428] Setting key d06498fea15aeaaebb1a49501a17ac89
I20260924 10:02:03.179811 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33893
--builtin_ntp_servers=127.13.162.20:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:02:03.180228 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.174778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:54357 (local address 127.13.162.62:33893)
0924 10:02:03.174937 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.174942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.176307 (+ 1365us) server_negotiation.cc:407] Connection header received
0924 10:02:03.178066 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.178071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.178134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.178217 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.179570 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.179575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.179578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.179593 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.179623 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.179633 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.179729 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.179901 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.179961 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:03.181145 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.175290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33893 (local address 127.13.162.2:54357)
0924 10:02:03.176152 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.176188 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.177818 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.178478 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.178485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.178581 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.179281 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.179283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.179320 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.179404 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.179714 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.179724 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.179732 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.179970 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.180259 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":247,"threads_started":1}
I20260924 10:02:03.182482 18930 heartbeater.cc:347] Connected to a master server at 127.13.162.62:33893
I20260924 10:02:03.182760 18930 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:03.183292 18930 heartbeater.cc:511] Master 127.13.162.62:33893 requested a full tablet report, sending...
I20260924 10:02:03.184646 18614 ts_manager.cc:194] Registered new tserver with Master: 52d665453478412ba2f1944379432aee (127.13.162.2:45357)
W20260924 10:02:03.493645 18933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:03.493963 18933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:03.494082 18933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:03.501762 18933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:03.501899 18933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:03.502099 18933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:03.517125 18933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:03.519052 18933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:03.521538 18933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:03.536759 18939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:03.536967 18941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:03.537585 18938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:03.537828 18933 server_base.cc:1034] running on GCE node
I20260924 10:02:03.538506 18933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:03.539502 18933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:03.540736 18933 hybrid_clock.cc:648] HybridClock initialized: now 1790244123540663 us; error 65 us; skew 500 ppm
I20260924 10:02:03.541113 18933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:03.543659 18933 webserver.cc:460] Webserver started at http://127.13.162.3:45221/ using document root <none> and password file <none>
I20260924 10:02:03.544349 18933 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:03.544466 18933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:03.544804 18933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:03.547747 18933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "5f340bb7721746468a92a7b04046f1be"
format_stamp: "Formatted at 2026-09-24 10:02:03 on dist-test-slave-jvsj"
server_key: "75b1b9dd7ecf5c9079af6ee7d97b121c"
server_key_iv: "264efa0f747032c05c500da7c3ea2d57"
server_key_version: "encryptionkey@0"
I20260924 10:02:03.548542 18933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "5f340bb7721746468a92a7b04046f1be"
format_stamp: "Formatted at 2026-09-24 10:02:03 on dist-test-slave-jvsj"
server_key: "75b1b9dd7ecf5c9079af6ee7d97b121c"
server_key_iv: "264efa0f747032c05c500da7c3ea2d57"
server_key_version: "encryptionkey@0"
I20260924 10:02:03.554098 18933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:03.558074 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:03.559739 18933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:03.559916 18933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "5f340bb7721746468a92a7b04046f1be"
format_stamp: "Formatted at 2026-09-24 10:02:03 on dist-test-slave-jvsj"
server_key: "75b1b9dd7ecf5c9079af6ee7d97b121c"
server_key_iv: "264efa0f747032c05c500da7c3ea2d57"
server_key_version: "encryptionkey@0"
I20260924 10:02:03.560158 18933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:03.566423 18933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:03.567270 18933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:03.567562 18933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:03.568542 18933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:03.569999 18933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:03.570092 18933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:03.570194 18933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:03.570258 18933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:03.600301 18933 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:36457
I20260924 10:02:03.600328 19059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:36457 every 8 connection(s)
I20260924 10:02:03.601732 18933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:03.608502 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 18933
I20260924 10:02:03.608680 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:03.609222 13960 external_mini_cluster.cc:1428] Setting key 5f9b93f754e576ba538544cdf3513836
I20260924 10:02:03.610487 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.604340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:36771 (local address 127.13.162.62:33893)
0924 10:02:03.604525 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.604529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.606099 (+ 1570us) server_negotiation.cc:407] Connection header received
0924 10:02:03.607893 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.607898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.607987 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.608084 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.609657 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.609662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.609666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.609683 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.609717 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.609733 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.609862 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.610115 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.610181 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:02:03.611670 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.604890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33893 (local address 127.13.162.3:36771)
0924 10:02:03.605937 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.605984 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.607606 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.608366 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.608378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.608515 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.609291 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.609294 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.609347 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.609466 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.609860 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.609872 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.609880 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.610177 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.610461 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":291,"threads_started":1}
I20260924 10:02:03.612888 19060 heartbeater.cc:347] Connected to a master server at 127.13.162.62:33893
I20260924 10:02:03.613209 19060 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:03.613768 19060 heartbeater.cc:511] Master 127.13.162.62:33893 requested a full tablet report, sending...
I20260924 10:02:03.615130 18614 ts_manager.cc:194] Registered new tserver with Master: 5f340bb7721746468a92a7b04046f1be (127.13.162.3:36457)
I20260924 10:02:03.624596 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:02:03.631057 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.628800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.13.162.62:33893)
0924 10:02:03.628977 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.628985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.629429 (+ 444us) server_negotiation.cc:407] Connection header received
0924 10:02:03.629583 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.629587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.629637 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.629713 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.630566 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.630572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.630577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.630596 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.630632 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.630647 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.630753 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.630793 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.630834 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:03.634902 18614 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:03.641672 18614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:03.660265 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.654572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44022)
0924 10:02:03.656335 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.656351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.656431 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.657480 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.657488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.657599 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.658397 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.658400 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.658428 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.658516 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.659669 (+ 1153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.659682 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.659692 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.659869 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.659950 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"thread_start_us":2251,"threads_started":1}
I20260924 10:02:03.660856 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.655086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.13.162.1:38035)
0924 10:02:03.656857 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.656865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.656911 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:02:03.656991 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.657000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.657259 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.658063 (+ 804us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.659433 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.659442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.659456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.659477 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.659529 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.659548 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.659675 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.660498 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.660593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1636,"thread_start_us":1533,"threads_started":1}
I20260924 10:02:03.661314 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.654472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:36457 (local address 127.0.0.1:38624)
0924 10:02:03.655605 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.655646 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.655788 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.658594 (+ 2806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.658598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.658620 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.659229 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.659233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.659248 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.659291 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.660839 (+ 1548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.660844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.660849 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.660980 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.661027 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":770,"threads_started":1}
I20260924 10:02:03.662952 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.654963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.13.162.3:36457)
0924 10:02:03.657399 (+ 2436us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.657407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.657450 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:02:03.657509 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.657516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.658401 (+ 885us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.659051 (+ 650us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.660328 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.660335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.660610 (+ 275us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.660632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.660691 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.660716 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.660883 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.660953 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.661839 (+ 886us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2281,"thread_start_us":2129,"threads_started":1}
I20260924 10:02:03.663406 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.657224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.13.162.2:45357)
0924 10:02:03.660084 (+ 2860us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.660100 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.660127 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:02:03.660190 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.660197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.660395 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.660536 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.662661 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.662666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.662676 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.662697 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.662741 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.662755 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.662907 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.662971 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.663063 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2707,"thread_start_us":2574,"threads_started":1}
I20260924 10:02:03.663461 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.656619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45357 (local address 127.0.0.1:57398)
0924 10:02:03.657677 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.657696 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.657797 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.661465 (+ 3668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.661469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.661492 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.662466 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.662469 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.662483 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.662523 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.662851 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.662856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.662860 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.663020 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.663064 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":785,"threads_started":1}
I20260924 10:02:03.664947 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 733889c6af6a48998c1c0ade59d6da74 (DEFAULT_TABLE table=test-table [id=0fd67424cd754007987d6741371311e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:03.666445 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733889c6af6a48998c1c0ade59d6da74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:03.666395 18995 tablet_service.cc:1467] Processing CreateTablet for tablet 733889c6af6a48998c1c0ade59d6da74 (DEFAULT_TABLE table=test-table [id=0fd67424cd754007987d6741371311e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:03.666894 18865 tablet_service.cc:1467] Processing CreateTablet for tablet 733889c6af6a48998c1c0ade59d6da74 (DEFAULT_TABLE table=test-table [id=0fd67424cd754007987d6741371311e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:03.667552 18995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733889c6af6a48998c1c0ade59d6da74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:03.668155 18865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733889c6af6a48998c1c0ade59d6da74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:03.683540 19078 tablet_bootstrap.cc:492] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Bootstrap starting.
I20260924 10:02:03.684298 19079 tablet_bootstrap.cc:492] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: Bootstrap starting.
I20260924 10:02:03.686254 19080 tablet_bootstrap.cc:492] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Bootstrap starting.
I20260924 10:02:03.687870 19079 tablet_bootstrap.cc:654] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:03.688829 19079 log.cc:824] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:03.689646 19080 tablet_bootstrap.cc:654] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:03.692229 19078 tablet_bootstrap.cc:654] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:03.692847 19080 log.cc:824] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:03.693387 19078 log.cc:824] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:03.695993 19079 tablet_bootstrap.cc:492] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: No bootstrap required, opened a new log
I20260924 10:02:03.696317 19079 ts_tablet_manager.cc:1397] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260924 10:02:03.700388 19080 tablet_bootstrap.cc:492] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: No bootstrap required, opened a new log
I20260924 10:02:03.700693 19080 ts_tablet_manager.cc:1397] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260924 10:02:03.706420 19079 raft_consensus.cc:348] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.707096 19079 raft_consensus.cc:374] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:03.707264 19079 raft_consensus.cc:729] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d22447558b4200ba9d554d1f2236d3, State: Initialized, Role: FOLLOWER
I20260924 10:02:03.708025 19079 consensus_queue.cc:260] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.709717 18800 heartbeater.cc:503] Master 127.13.162.62:33893 was elected leader, sending a full tablet report...
I20260924 10:02:03.709954 19079 ts_tablet_manager.cc:1428] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 10:02:03.711760 19080 raft_consensus.cc:348] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.712111 19080 raft_consensus.cc:374] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:03.712208 19080 raft_consensus.cc:729] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f340bb7721746468a92a7b04046f1be, State: Initialized, Role: FOLLOWER
I20260924 10:02:03.712756 19080 consensus_queue.cc:260] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.714128 19060 heartbeater.cc:503] Master 127.13.162.62:33893 was elected leader, sending a full tablet report...
I20260924 10:02:03.714242 19080 ts_tablet_manager.cc:1428] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260924 10:02:03.718400 19078 tablet_bootstrap.cc:492] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: No bootstrap required, opened a new log
I20260924 10:02:03.718721 19078 ts_tablet_manager.cc:1397] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.016s
I20260924 10:02:03.728658 19078 raft_consensus.cc:348] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.729024 19078 raft_consensus.cc:374] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:03.729096 19078 raft_consensus.cc:729] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d665453478412ba2f1944379432aee, State: Initialized, Role: FOLLOWER
I20260924 10:02:03.729666 19078 consensus_queue.cc:260] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.730700 18930 heartbeater.cc:503] Master 127.13.162.62:33893 was elected leader, sending a full tablet report...
I20260924 10:02:03.731402 19078 ts_tablet_manager.cc:1428] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260924 10:02:03.758198 19086 raft_consensus.cc:482] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:03.758618 19086 raft_consensus.cc:504] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.760332 19086 leader_election.cc:288] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f340bb7721746468a92a7b04046f1be (127.13.162.3:36457), 90d22447558b4200ba9d554d1f2236d3 (127.13.162.1:38035)
I20260924 10:02:03.763751 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.760711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:50197 (local address 127.13.162.3:36457)
0924 10:02:03.760876 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.760882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.761739 (+ 857us) server_negotiation.cc:407] Connection header received
0924 10:02:03.761897 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.761902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.761965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.762068 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.763177 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.763183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.763187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.763207 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.763246 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.763261 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.763395 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.763444 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.763501 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:02:03.763767 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.760711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:59267 (local address 127.13.162.1:38035)
0924 10:02:03.760876 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.760882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.761739 (+ 857us) server_negotiation.cc:407] Connection header received
0924 10:02:03.761897 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.761902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.762010 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.762104 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.763200 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.763205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.763209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.763226 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.763260 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.763273 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.763422 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.763466 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.763514 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:03.764207 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.760789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.13.162.2:59267)
0924 10:02:03.761609 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.761633 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.761779 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.762110 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.762115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.762139 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.763015 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.763018 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.763031 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.763061 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.763355 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.763359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.763364 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.763531 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.763598 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":18,"spinlock_wait_cycles":16640,"thread_start_us":432,"threads_started":1}
I20260924 10:02:03.764230 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.760792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:36457 (local address 127.13.162.2:50197)
0924 10:02:03.761643 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.761658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:03.761779 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:03.762110 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:03.762115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:03.762139 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:03.762927 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:03.762931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:03.762952 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:03.763001 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:03.763360 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:03.763364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:03.763368 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:02:03.763523 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:02:03.763585 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":28,"spinlock_wait_cycles":15616,"thread_start_us":433,"threads_started":1}
I20260924 10:02:03.765179 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733889c6af6a48998c1c0ade59d6da74" candidate_uuid: "52d665453478412ba2f1944379432aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d22447558b4200ba9d554d1f2236d3" is_pre_election: true
I20260924 10:02:03.765249 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733889c6af6a48998c1c0ade59d6da74" candidate_uuid: "52d665453478412ba2f1944379432aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f340bb7721746468a92a7b04046f1be" is_pre_election: true
I20260924 10:02:03.765642 18755 raft_consensus.cc:2454] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52d665453478412ba2f1944379432aee in term 0.
I20260924 10:02:03.765715 19015 raft_consensus.cc:2454] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52d665453478412ba2f1944379432aee in term 0.
I20260924 10:02:03.766249 18818 leader_election.cc:302] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d665453478412ba2f1944379432aee, 90d22447558b4200ba9d554d1f2236d3; no voters:
I20260924 10:02:03.766655 19086 raft_consensus.cc:2781] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:03.766779 19086 raft_consensus.cc:482] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:03.766871 19086 raft_consensus.cc:3037] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:03.769034 19086 raft_consensus.cc:504] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.769590 19086 leader_election.cc:288] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [CANDIDATE]: Term 1 election: Requested vote from peers 5f340bb7721746468a92a7b04046f1be (127.13.162.3:36457), 90d22447558b4200ba9d554d1f2236d3 (127.13.162.1:38035)
I20260924 10:02:03.770027 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733889c6af6a48998c1c0ade59d6da74" candidate_uuid: "52d665453478412ba2f1944379432aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f340bb7721746468a92a7b04046f1be"
I20260924 10:02:03.770076 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733889c6af6a48998c1c0ade59d6da74" candidate_uuid: "52d665453478412ba2f1944379432aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d22447558b4200ba9d554d1f2236d3"
I20260924 10:02:03.770193 19015 raft_consensus.cc:3037] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:03.770264 18755 raft_consensus.cc:3037] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 0 FOLLOWER]: Advancing to term 1
W20260924 10:02:03.770918 18801 tablet.cc:2369] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:02:03.772683 19015 raft_consensus.cc:2454] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52d665453478412ba2f1944379432aee in term 1.
I20260924 10:02:03.773051 18821 leader_election.cc:302] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d665453478412ba2f1944379432aee, 5f340bb7721746468a92a7b04046f1be; no voters:
I20260924 10:02:03.773312 19086 raft_consensus.cc:2781] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:03.773445 18755 raft_consensus.cc:2454] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52d665453478412ba2f1944379432aee in term 1.
I20260924 10:02:03.773752 19086 raft_consensus.cc:686] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 1 LEADER]: Becoming Leader. State: Replica: 52d665453478412ba2f1944379432aee, State: Running, Role: LEADER
I20260924 10:02:03.774269 19086 consensus_queue.cc:237] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } }
I20260924 10:02:03.778671 18613 catalog_manager.cc:5495] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee reported cstate change: term changed from 0 to 1, leader changed from <none> to 52d665453478412ba2f1944379432aee (127.13.162.2). New cstate: current_term: 1 leader_uuid: "52d665453478412ba2f1944379432aee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52d665453478412ba2f1944379432aee" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45357 } health_report { overall_health: HEALTHY } } }
I20260924 10:02:03.818742 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.816708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.13.162.2:45357)
0924 10:02:03.816890 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.816896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.816916 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:02:03.817172 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.817176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.817265 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.817351 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.818160 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.818164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.818168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.818182 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.818219 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.818231 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.818357 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.818507 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.818555 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:02:03.826761 18755 raft_consensus.cc:1261] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 1 FOLLOWER]: Refusing update from remote peer 52d665453478412ba2f1944379432aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:03.826764 19015 raft_consensus.cc:1261] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 1 FOLLOWER]: Refusing update from remote peer 52d665453478412ba2f1944379432aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:03.827690 19089 consensus_queue.cc:1035] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d22447558b4200ba9d554d1f2236d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:03.828022 19086 consensus_queue.cc:1035] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f340bb7721746468a92a7b04046f1be" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:03.844540 19094 mvcc.cc:205] Tried to move back new op lower bound from 7332839931184148480 to 7332839930983362560. Current Snapshot: MvccSnapshot[applied={T|T < 7332839931184148480}]
I20260924 10:02:03.851198 19093 mvcc.cc:205] Tried to move back new op lower bound from 7332839931184148480 to 7332839930983362560. Current Snapshot: MvccSnapshot[applied={T|T < 7332839931184148480 or (T in {7332839931184148480})}]
I20260924 10:02:03.852681 18613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:03.852918 18613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:03.856798 18613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9055a3adcad241cfb7cf02721823870a: Sending DeleteTablet for 3 replicas of tablet 733889c6af6a48998c1c0ade59d6da74
I20260924 10:02:03.858042 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 733889c6af6a48998c1c0ade59d6da74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:03 UTC) from {username='slave'} at 127.0.0.1:44022
I20260924 10:02:03.858369 18995 tablet_service.cc:1514] Processing DeleteTablet for tablet 733889c6af6a48998c1c0ade59d6da74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:03 UTC) from {username='slave'} at 127.0.0.1:38624
I20260924 10:02:03.858366 18865 tablet_service.cc:1514] Processing DeleteTablet for tablet 733889c6af6a48998c1c0ade59d6da74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:03 UTC) from {username='slave'} at 127.0.0.1:57398
I20260924 10:02:03.860405 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18673
I20260924 10:02:03.871251 19102 tablet_replica.cc:330] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: stopping tablet replica
I20260924 10:02:03.871750 19102 raft_consensus.cc:2229] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:03.872280 19102 raft_consensus.cc:2258] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:03.874660 19102 ts_tablet_manager.cc:1905] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:03.881873 19102 ts_tablet_manager.cc:1918] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:03.882061 19102 log.cc:1196] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/733889c6af6a48998c1c0ade59d6da74
I20260924 10:02:03.882849 19103 tablet_replica.cc:330] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: stopping tablet replica
I20260924 10:02:03.883351 19103 raft_consensus.cc:2229] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:03.884040 19103 raft_consensus.cc:2258] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:03.884783 19102 ts_tablet_manager.cc:1939] T 733889c6af6a48998c1c0ade59d6da74 P 5f340bb7721746468a92a7b04046f1be: Deleting consensus metadata
I20260924 10:02:03.885990 19103 ts_tablet_manager.cc:1905] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:03.887008 18601 catalog_manager.cc:4841] TS 5f340bb7721746468a92a7b04046f1be (127.13.162.3:36457): tablet 733889c6af6a48998c1c0ade59d6da74 (table test-table [id=0fd67424cd754007987d6741371311e4]) successfully deleted
I20260924 10:02:03.887360 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.862151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.13.162.1:38035)
0924 10:02:03.862358 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.862365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.884585 (+ 22220us) server_negotiation.cc:407] Connection header received
0924 10:02:03.885096 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.885102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.885212 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.885289 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.886455 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.886461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.886465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.886495 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.886535 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.886552 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.886652 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.887006 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.887072 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:02:03.888195 19101 tablet_replica.cc:330] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3: stopping tablet replica
I20260924 10:02:03.888614 19101 raft_consensus.cc:2229] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:03.888634 18795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:03.894232 19103 ts_tablet_manager.cc:1918] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:03.894449 19103 log.cc:1196] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/733889c6af6a48998c1c0ade59d6da74
I20260924 10:02:03.895022 19103 ts_tablet_manager.cc:1939] T 733889c6af6a48998c1c0ade59d6da74 P 52d665453478412ba2f1944379432aee: Deleting consensus metadata
I20260924 10:02:03.896814 18600 catalog_manager.cc:4841] TS 52d665453478412ba2f1944379432aee (127.13.162.2:45357): tablet 733889c6af6a48998c1c0ade59d6da74 (table test-table [id=0fd67424cd754007987d6741371311e4]) successfully deleted
I20260924 10:02:03.975809 19101 raft_consensus.cc:2258] T 733889c6af6a48998c1c0ade59d6da74 P 90d22447558b4200ba9d554d1f2236d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:03.976446 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18673
W20260924 10:02:03.989933 18598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:38035 (error 108)
I20260924 10:02:03.990164 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18803
I20260924 10:02:03.992276 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.991752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44040)
0924 10:02:03.992008 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:03.992122 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260924 10:02:03.992517 18598 catalog_manager.cc:4584] TS 90d22447558b4200ba9d554d1f2236d3 (127.13.162.1:38035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 733889c6af6a48998c1c0ade59d6da74: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
I20260924 10:02:03.993327 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:03.990977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.13.162.2:45357)
0924 10:02:03.991209 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:03.991215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:03.991276 (+ 61us) server_negotiation.cc:407] Connection header received
0924 10:02:03.991473 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:03.991479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:03.991547 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:03.991663 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:03.992603 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:03.992613 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:03.992617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:03.992636 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:03.992674 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:03.992690 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:03.992820 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:03.992936 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 10:02:03.992998 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:02:03.994122 18925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:04.045984 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.045460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44054)
0924 10:02:04.045672 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.045796 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:02:04.046972 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.046629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44058)
0924 10:02:04.046761 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.046839 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 10:02:04.100863 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18803
I20260924 10:02:04.107357 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.106905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44066)
0924 10:02:04.107088 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.107194 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:02:04.108242 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.107966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44072)
0924 10:02:04.108074 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.108136 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 10:02:04.117460 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18933
I20260924 10:02:04.119905 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.118222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.13.162.3:36457)
0924 10:02:04.118401 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:04.118405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:04.118424 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:02:04.118551 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:04.118555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:04.118626 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:04.118702 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:04.119453 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:04.119458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:04.119461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:04.119476 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:04.119507 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:04.119518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:04.119628 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:04.119661 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 10:02:04.119701 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:02:04.120493 19055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:04.188787 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.188215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44080)
0924 10:02:04.188500 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.188618 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 10:02:04.189805 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.189392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38035 (local address 127.0.0.1:44082)
0924 10:02:04.189586 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.189668 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 10:02:04.206334 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18933
I20260924 10:02:04.218918 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 18583
I20260924 10:02:04.219964 18644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:04.321632 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 18583
2026-09-24T10:02:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2497 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 10:02:04.357682 13960 mini_oidc.cc:152] Starting JWKS server
I20260924 10:02:04.360280 13960 webserver.cc:460] Webserver started at https://127.0.0.1:36685/ using document root <none> and password file <none>
I20260924 10:02:04.360572 13960 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:02:04.361178 13960 webserver.cc:460] Webserver started at http://127.0.0.1:43253/ using document root <none> and password file <none>
2026-09-24T10:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:04Z Disabled control of system clock
I20260924 10:02:04.383903 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45477
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:45477
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36685/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/testchainca.cert with env {}
W20260924 10:02:04.696817 19115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:04.697147 19115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:04.697281 19115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:04.704746 19115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:04.704865 19115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:04.704926 19115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:04.705003 19115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:04.705070 19115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:02:04.705149 19115 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:02:04.705204 19115 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36685/jwks.json
I20260924 10:02:04.715237 19115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:45477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36685/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:04.716559 19115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:04.717890 19115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:04.729144 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:04.729182 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:04.729205 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:04.730279 19115 server_base.cc:1034] running on GCE node
I20260924 10:02:04.731184 19115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:04.732699 19115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:04.733928 19115 hybrid_clock.cc:648] HybridClock initialized: now 1790244124733877 us; error 31 us; skew 500 ppm
I20260924 10:02:04.734289 19115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:04.736536 19115 webserver.cc:460] Webserver started at http://127.13.162.62:35045/ using document root <none> and password file <none>
I20260924 10:02:04.737239 19115 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:04.737345 19115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:04.737650 19115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:04.740564 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "48fa9e55c7f44a7f8a9ec2885b525631"
format_stamp: "Formatted at 2026-09-24 10:02:04 on dist-test-slave-jvsj"
server_key: "5057bb4e0ee312f94cd850aa397d7b81"
server_key_iv: "4419b41566d3b364c24a484dd01d68a0"
server_key_version: "encryptionkey@0"
I20260924 10:02:04.741318 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "48fa9e55c7f44a7f8a9ec2885b525631"
format_stamp: "Formatted at 2026-09-24 10:02:04 on dist-test-slave-jvsj"
server_key: "5057bb4e0ee312f94cd850aa397d7b81"
server_key_iv: "4419b41566d3b364c24a484dd01d68a0"
server_key_version: "encryptionkey@0"
I20260924 10:02:04.746222 19115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:02:04.749593 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:04.750983 19115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:04.751142 19115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "48fa9e55c7f44a7f8a9ec2885b525631"
format_stamp: "Formatted at 2026-09-24 10:02:04 on dist-test-slave-jvsj"
server_key: "5057bb4e0ee312f94cd850aa397d7b81"
server_key_iv: "4419b41566d3b364c24a484dd01d68a0"
server_key_version: "encryptionkey@0"
I20260924 10:02:04.751330 19115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:04.778525 19115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:04.779347 19115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:04.779647 19115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:04.796182 19115 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:45477
I20260924 10:02:04.796205 19180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:45477 every 8 connection(s)
I20260924 10:02:04.797475 19115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:04.799588 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19115
I20260924 10:02:04.799801 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:04.800554 13960 external_mini_cluster.cc:1428] Setting key 7a7d916424c938d366f27a80135751ab
I20260924 10:02:04.802476 19181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:04.813620 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.802219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.13.162.62:45477)
0924 10:02:04.803159 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:04.803171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:04.803212 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:02:04.804789 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:04.804810 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:04.805287 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:04.805626 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:04.806197 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:04.807560 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:04.809383 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:04.810026 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:04.810157 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:04.811352 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:04.811359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:04.811376 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:04.811945 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:04.812019 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:04.812039 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:04.812138 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:04.812343 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 10:02:04.812677 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":318,"threads_started":1}
I20260924 10:02:04.814532 19181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631: Bootstrap starting.
I20260924 10:02:04.817341 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:02:04.817739 19181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:04.818691 19181 log.cc:824] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:04.821341 19181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631: No bootstrap required, opened a new log
I20260924 10:02:04.828217 19181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45477 } }
I20260924 10:02:04.828601 19181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:04.828696 19181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fa9e55c7f44a7f8a9ec2885b525631, State: Initialized, Role: FOLLOWER
I20260924 10:02:04.829231 19181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45477 } }
I20260924 10:02:04.829430 19181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:04.829551 19181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:04.829689 19181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:04.831800 19181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45477 } }
I20260924 10:02:04.832273 19181 leader_election.cc:302] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48fa9e55c7f44a7f8a9ec2885b525631; no voters:
I20260924 10:02:04.832772 19181 leader_election.cc:288] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:04.832904 19186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:04.834124 19186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [term 1 LEADER]: Becoming Leader. State: Replica: 48fa9e55c7f44a7f8a9ec2885b525631, State: Running, Role: LEADER
I20260924 10:02:04.834649 19186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45477 } }
I20260924 10:02:04.835121 19181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:04.837575 19188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48fa9e55c7f44a7f8a9ec2885b525631. Latest consensus state: current_term: 1 leader_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45477 } } }
I20260924 10:02:04.837970 19188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:04.838433 19187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fa9e55c7f44a7f8a9ec2885b525631" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45477 } } }
I20260924 10:02:04.838742 19187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:04.839425 19194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:04.844233 19194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:04.853989 19194 catalog_manager.cc:1347] Generated new cluster ID: ba73eef7b5694a43acb118a4c08c015c
I20260924 10:02:04.854125 19194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:04.865882 19194 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:04.867134 19194 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:04.877321 19194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48fa9e55c7f44a7f8a9ec2885b525631: Generated new TSK 0
I20260924 10:02:04.878402 19194 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:02:04.918045 19211 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 10:02:04.918192 19211 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:02:04.919391 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:04.898420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.13.162.62:45477)
0924 10:02:04.898603 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:04.898613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:04.899026 (+ 413us) server_negotiation.cc:407] Connection header received
0924 10:02:04.899301 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:04.899307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:04.899392 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:04.899550 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:02:04.899996 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:04.901104 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:04.902447 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:04.903166 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:04.903290 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:04.903734 (+ 444us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:02:04.918715 (+ 14981us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:02:04.919079 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 10:02:04.919515 19184 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 10:02:04.921044 19210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:04.898247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45477 (local address 127.0.0.1:52246)
0924 10:02:04.898932 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:04.898951 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:04.899093 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:04.899571 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:04.899574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:04.899583 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:02:04.899877 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:04.899885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:04.901212 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:04.901215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:04.902291 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:04.902301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:04.903316 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:04.903320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:04.903435 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:04.903486 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:02:04.918999 (+ 15513us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:02:04.919325 (+ 326us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.162.62:45477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":380,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.162.62:45477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 10:02:04.922348 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19115
I20260924 10:02:04.923436 19176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:05.013129 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19115
2026-09-24T10:02:05Z chronyd exiting
I20260924 10:02:05.059820 13960 test_util.cc:175] -----------------------------------------------
I20260924 10:02:05.059978 13960 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244085964582-13960-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 10:02:05.063730 13960 mini_oidc.cc:152] Starting JWKS server
I20260924 10:02:05.065593 13960 webserver.cc:460] Webserver started at https://127.0.0.1:43425/ using document root <none> and password file <none>
I20260924 10:02:05.065814 13960 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:02:05.066520 13960 webserver.cc:460] Webserver started at http://127.0.0.1:35365/ using document root <none> and password file <none>
2026-09-24T10:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:05Z Disabled control of system clock
I20260924 10:02:05.088447 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36985
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36985
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/testchainca.cert with env {}
W20260924 10:02:05.395332 19218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:05.395629 19218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:05.395761 19218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:05.403216 19218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:05.403333 19218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:05.403393 19218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:05.403468 19218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:05.403525 19218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:02:05.403606 19218 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:02:05.403653 19218 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43425/jwks.json
I20260924 10:02:05.413455 19218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:05.414727 19218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:05.415972 19218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:05.427034 19223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:05.426929 19224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:05.426990 19226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:05.427980 19218 server_base.cc:1034] running on GCE node
I20260924 10:02:05.428699 19218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:05.430128 19218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:05.431341 19218 hybrid_clock.cc:648] HybridClock initialized: now 1790244125431285 us; error 49 us; skew 500 ppm
I20260924 10:02:05.431694 19218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:05.433880 19218 webserver.cc:460] Webserver started at http://127.13.162.62:37885/ using document root <none> and password file <none>
I20260924 10:02:05.434604 19218 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:05.434706 19218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:05.435009 19218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:05.437952 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "667618e97886420497dce48cd4606b8d"
format_stamp: "Formatted at 2026-09-24 10:02:05 on dist-test-slave-jvsj"
server_key: "1dda4a1af73276e4347ec1fd36ae7357"
server_key_iv: "9bacc4ae7b419a1d7310ced7abfa572c"
server_key_version: "encryptionkey@0"
I20260924 10:02:05.438735 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "667618e97886420497dce48cd4606b8d"
format_stamp: "Formatted at 2026-09-24 10:02:05 on dist-test-slave-jvsj"
server_key: "1dda4a1af73276e4347ec1fd36ae7357"
server_key_iv: "9bacc4ae7b419a1d7310ced7abfa572c"
server_key_version: "encryptionkey@0"
I20260924 10:02:05.443734 19218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 10:02:05.447317 19232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:05.448810 19218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:05.448975 19218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "667618e97886420497dce48cd4606b8d"
format_stamp: "Formatted at 2026-09-24 10:02:05 on dist-test-slave-jvsj"
server_key: "1dda4a1af73276e4347ec1fd36ae7357"
server_key_iv: "9bacc4ae7b419a1d7310ced7abfa572c"
server_key_version: "encryptionkey@0"
I20260924 10:02:05.449174 19218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:05.463672 19218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:05.464486 19218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:05.464768 19218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:05.481685 19218 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:36985
I20260924 10:02:05.481704 19283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:36985 every 8 connection(s)
I20260924 10:02:05.483011 19218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:05.486618 19284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:05.492971 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19218
I20260924 10:02:05.493178 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:05.493957 13960 external_mini_cluster.cc:1428] Setting key 37f06030dd185cce1e54ebd71c84597d
I20260924 10:02:05.497469 19284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d: Bootstrap starting.
I20260924 10:02:05.500597 19284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:05.501627 19284 log.cc:824] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:05.504391 19284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d: No bootstrap required, opened a new log
I20260924 10:02:05.507428 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:05.495942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.13.162.62:36985)
0924 10:02:05.496860 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:05.496877 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:05.496932 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:02:05.498656 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:05.498680 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:05.499185 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:05.499609 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:05.500199 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:05.501578 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:05.503164 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:05.503819 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:05.503987 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:05.505083 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:05.505091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:05.505106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:05.505642 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:05.505722 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:05.505746 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:05.505861 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:05.506095 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 10:02:05.506442 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":290,"threads_started":1}
I20260924 10:02:05.511225 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:02:05.512429 19284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667618e97886420497dce48cd4606b8d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36985 } }
I20260924 10:02:05.512773 19284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:05.512854 19284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667618e97886420497dce48cd4606b8d, State: Initialized, Role: FOLLOWER
I20260924 10:02:05.513414 19284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667618e97886420497dce48cd4606b8d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36985 } }
I20260924 10:02:05.513659 19284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:05.513782 19284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:05.513931 19284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:05.516175 19284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667618e97886420497dce48cd4606b8d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36985 } }
I20260924 10:02:05.516645 19284 leader_election.cc:302] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667618e97886420497dce48cd4606b8d; no voters:
I20260924 10:02:05.517293 19284 leader_election.cc:288] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:05.517412 19289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:05.518007 19289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [term 1 LEADER]: Becoming Leader. State: Replica: 667618e97886420497dce48cd4606b8d, State: Running, Role: LEADER
I20260924 10:02:05.518584 19289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667618e97886420497dce48cd4606b8d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36985 } }
I20260924 10:02:05.519451 19284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:05.521266 19291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667618e97886420497dce48cd4606b8d. Latest consensus state: current_term: 1 leader_uuid: "667618e97886420497dce48cd4606b8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667618e97886420497dce48cd4606b8d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36985 } } }
I20260924 10:02:05.521663 19291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:05.522379 19290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "667618e97886420497dce48cd4606b8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667618e97886420497dce48cd4606b8d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36985 } } }
I20260924 10:02:05.522707 19290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:05.522585 19298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:05.527949 19298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:05.536550 19298 catalog_manager.cc:1347] Generated new cluster ID: c9c7d28166c64c3093785b0cb64949e2
I20260924 10:02:05.536651 19298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:05.548095 19298 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:05.549288 19298 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:05.555342 19298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 667618e97886420497dce48cd4606b8d: Generated new TSK 0
I20260924 10:02:05.556116 19298 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:02:05.600795 19314 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 10:02:05.600904 19314 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:02:05.602236 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:05.575046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.13.162.62:36985)
0924 10:02:05.575235 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:05.575243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:05.575762 (+ 519us) server_negotiation.cc:407] Connection header received
0924 10:02:05.575981 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:05.575986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:05.576062 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:05.576192 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:02:05.576748 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:05.577540 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:05.579516 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:05.580004 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:05.580100 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:05.580661 (+ 561us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:02:05.601553 (+ 20892us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:02:05.601961 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:02:05.602388 19287 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 10:02:05.602411 19313 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:05.575026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36985 (local address 127.0.0.1:54680)
0924 10:02:05.575642 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:05.575661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:05.575818 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:05.576227 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:05.576232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:05.576245 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:02:05.576570 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:05.576583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:05.577677 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:05.577683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:05.579363 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:05.579379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:05.580158 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:05.580165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:05.580352 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:05.580396 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:02:05.601836 (+ 21440us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:02:05.602066 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.162.62:36985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":307,"threads_started":1}
I20260924 10:02:05.604326 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19218
I20260924 10:02:05.605587 19279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:05.696137 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19218
2026-09-24T10:02:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (684 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 10:02:05.747774 13960 mini_oidc.cc:152] Starting JWKS server
I20260924 10:02:05.749536 13960 webserver.cc:460] Webserver started at https://127.0.0.1:38631/ using document root <none> and password file <none>
I20260924 10:02:05.749778 13960 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:02:05.750352 13960 webserver.cc:460] Webserver started at http://127.0.0.1:43341/ using document root <none> and password file <none>
2026-09-24T10:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:05Z Disabled control of system clock
I20260924 10:02:05.772840 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:33513
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:33513
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38631/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:02:06.078259 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:06.078567 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:06.078660 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:06.085747 19321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:06.085865 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:06.085901 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:06.085954 19321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:06.085999 19321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:02:06.086076 19321 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:02:06.086115 19321 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38631/jwks.json
I20260924 10:02:06.095149 19321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:33513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:33513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38631/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:06.096388 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:06.098229 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:06.109742 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:06.109776 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:06.109707 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:06.110622 19321 server_base.cc:1034] running on GCE node
I20260924 10:02:06.111399 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:06.112788 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:06.113973 19321 hybrid_clock.cc:648] HybridClock initialized: now 1790244126113919 us; error 47 us; skew 500 ppm
I20260924 10:02:06.114343 19321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:06.119570 19321 webserver.cc:460] Webserver started at http://127.13.162.62:38711/ using document root <none> and password file <none>
I20260924 10:02:06.120271 19321 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:06.120361 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:06.120673 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:06.123622 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "726e4b7a4a0b437f99507109076a1894"
format_stamp: "Formatted at 2026-09-24 10:02:06 on dist-test-slave-jvsj"
server_key: "d8e9e082d513deb23bcd5c20868e80ea"
server_key_iv: "67357f8b33da5bb0c1103e99a679e163"
server_key_version: "encryptionkey@0"
I20260924 10:02:06.124377 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "726e4b7a4a0b437f99507109076a1894"
format_stamp: "Formatted at 2026-09-24 10:02:06 on dist-test-slave-jvsj"
server_key: "d8e9e082d513deb23bcd5c20868e80ea"
server_key_iv: "67357f8b33da5bb0c1103e99a679e163"
server_key_version: "encryptionkey@0"
I20260924 10:02:06.129325 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:06.132702 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:06.134027 19321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:02:06.134198 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "726e4b7a4a0b437f99507109076a1894"
format_stamp: "Formatted at 2026-09-24 10:02:06 on dist-test-slave-jvsj"
server_key: "d8e9e082d513deb23bcd5c20868e80ea"
server_key_iv: "67357f8b33da5bb0c1103e99a679e163"
server_key_version: "encryptionkey@0"
I20260924 10:02:06.134408 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:06.147536 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:06.148350 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:06.148633 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:06.165892 19321 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:33513
I20260924 10:02:06.165911 19386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:33513 every 8 connection(s)
I20260924 10:02:06.167331 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:06.169272 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19321
I20260924 10:02:06.169512 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:06.170262 13960 external_mini_cluster.cc:1428] Setting key f2c3caa8ff39f49811e7760aaca4aac0
I20260924 10:02:06.172145 19387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:06.182833 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:06.172213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.13.162.62:33513)
0924 10:02:06.173168 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:06.173179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:06.173219 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:02:06.174746 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:06.174767 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:06.175224 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:06.175594 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:06.176133 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.177443 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.178730 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.179361 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.179529 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:06.180639 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:06.180647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:06.180660 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:06.181198 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:06.181278 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:06.181299 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:06.181400 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:06.181610 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 10:02:06.181884 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260924 10:02:06.183241 19387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894: Bootstrap starting.
I20260924 10:02:06.186211 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:02:06.186264 19387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:06.187222 19387 log.cc:824] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:06.190054 19387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894: No bootstrap required, opened a new log
I20260924 10:02:06.197167 19387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726e4b7a4a0b437f99507109076a1894" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33513 } }
I20260924 10:02:06.197515 19387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:06.197608 19387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726e4b7a4a0b437f99507109076a1894, State: Initialized, Role: FOLLOWER
I20260924 10:02:06.198148 19387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726e4b7a4a0b437f99507109076a1894" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33513 } }
I20260924 10:02:06.198374 19387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:06.198478 19387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:06.198606 19387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:06.200875 19387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726e4b7a4a0b437f99507109076a1894" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33513 } }
I20260924 10:02:06.201371 19387 leader_election.cc:302] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 726e4b7a4a0b437f99507109076a1894; no voters:
I20260924 10:02:06.202016 19387 leader_election.cc:288] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:06.202185 19392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:06.202841 19392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [term 1 LEADER]: Becoming Leader. State: Replica: 726e4b7a4a0b437f99507109076a1894, State: Running, Role: LEADER
I20260924 10:02:06.203315 19392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726e4b7a4a0b437f99507109076a1894" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33513 } }
I20260924 10:02:06.204061 19387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:06.206198 19394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 726e4b7a4a0b437f99507109076a1894. Latest consensus state: current_term: 1 leader_uuid: "726e4b7a4a0b437f99507109076a1894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726e4b7a4a0b437f99507109076a1894" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33513 } } }
I20260924 10:02:06.206264 19393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "726e4b7a4a0b437f99507109076a1894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726e4b7a4a0b437f99507109076a1894" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33513 } } }
I20260924 10:02:06.206579 19394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:06.206624 19393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:06.207703 19397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:06.213217 19397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:06.222549 19397 catalog_manager.cc:1347] Generated new cluster ID: ff93dec914d94378b3c493c17ca97d4c
I20260924 10:02:06.222656 19397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:06.234573 19397 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:06.236272 19397 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:06.261466 19397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 726e4b7a4a0b437f99507109076a1894: Generated new TSK 0
I20260924 10:02:06.262387 19397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:02:06.294205 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:06.285866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.13.162.62:33513)
0924 10:02:06.286063 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:06.286070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:06.286490 (+ 420us) server_negotiation.cc:407] Connection header received
0924 10:02:06.286716 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:06.286721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:06.286820 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:06.287006 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:02:06.287432 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.288663 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.289951 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.290677 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.290795 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:06.291242 (+ 447us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:02:06.293548 (+ 2306us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:02:06.293874 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:02:06.294360 19390 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 10:02:06.295843 19416 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:06.285814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33513 (local address 127.0.0.1:55886)
0924 10:02:06.286400 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:06.286414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:06.286534 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:06.287015 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:06.287018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:06.287027 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:02:06.287296 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:06.287304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.288790 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.288793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:06.289813 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:06.289823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.290833 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.290837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:06.290956 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:06.290987 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:02:06.293757 (+ 2770us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:02:06.295435 (+ 1678us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.162.62:33513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":1384,"thread_start_us":298,"threads_started":1}
I20260924 10:02:06.296778 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19321
I20260924 10:02:06.297955 19382 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 10:02:06.305583 19417 webserver.cc:533] Webserver: sslize failed:
E20260924 10:02:06.305684 19417 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:02:06.389258 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19321
2026-09-24T10:02:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (693 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T10:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:06Z Disabled control of system clock
I20260924 10:02:06.464094 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:43537
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:43537
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/testchainca.cert with env {}
W20260924 10:02:06.772019 19422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:06.772301 19422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:06.772397 19422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:06.779419 19422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:06.779527 19422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:06.779564 19422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:06.779618 19422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:06.779659 19422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:02:06.779711 19422 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:02:06.779748 19422 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 10:02:06.789384 19422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:43537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:06.790655 19422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:06.791894 19422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:06.803251 19427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:06.803325 19430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:06.803287 19428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:06.804015 19422 server_base.cc:1034] running on GCE node
I20260924 10:02:06.804754 19422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:06.806137 19422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:06.807333 19422 hybrid_clock.cc:648] HybridClock initialized: now 1790244126807277 us; error 47 us; skew 500 ppm
I20260924 10:02:06.807668 19422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:06.809732 19422 webserver.cc:460] Webserver started at http://127.13.162.62:42883/ using document root <none> and password file <none>
I20260924 10:02:06.810429 19422 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:06.810529 19422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:06.810814 19422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:06.813480 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "e30f3982063e49e382c3bf0eeae0e3ec"
format_stamp: "Formatted at 2026-09-24 10:02:06 on dist-test-slave-jvsj"
server_key: "c46f8ec806c4d739453cb54bfc58b92c"
server_key_iv: "7b3092b22d0a71d285acc0867491f915"
server_key_version: "encryptionkey@0"
I20260924 10:02:06.814224 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "e30f3982063e49e382c3bf0eeae0e3ec"
format_stamp: "Formatted at 2026-09-24 10:02:06 on dist-test-slave-jvsj"
server_key: "c46f8ec806c4d739453cb54bfc58b92c"
server_key_iv: "7b3092b22d0a71d285acc0867491f915"
server_key_version: "encryptionkey@0"
I20260924 10:02:06.819034 19422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:02:06.822288 19436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:06.823764 19422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:06.823913 19422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "e30f3982063e49e382c3bf0eeae0e3ec"
format_stamp: "Formatted at 2026-09-24 10:02:06 on dist-test-slave-jvsj"
server_key: "c46f8ec806c4d739453cb54bfc58b92c"
server_key_iv: "7b3092b22d0a71d285acc0867491f915"
server_key_version: "encryptionkey@0"
I20260924 10:02:06.824105 19422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:06.835606 19422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:06.836311 19422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:06.836544 19422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:06.852392 19422 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:43537
I20260924 10:02:06.852418 19487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:43537 every 8 connection(s)
I20260924 10:02:06.853698 19422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:06.857435 19488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:06.861038 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19422
I20260924 10:02:06.861205 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:06.861812 13960 external_mini_cluster.cc:1428] Setting key ee45a4e22ceefd136f169f61d6729306
I20260924 10:02:06.867990 19488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec: Bootstrap starting.
I20260924 10:02:06.870944 19488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:06.871958 19488 log.cc:824] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:06.875351 19488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec: No bootstrap required, opened a new log
I20260924 10:02:06.875553 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:06.863686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.13.162.62:43537)
0924 10:02:06.864633 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:06.864645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:06.864688 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:02:06.866811 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:06.866836 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:06.867310 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:06.867671 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:06.868222 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.869509 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.870775 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.871519 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.871677 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:06.872919 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:06.872927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:06.872944 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:06.873562 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:06.873660 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:06.873685 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:06.873803 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:06.873994 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:02:06.874306 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":304,"threads_started":1}
I20260924 10:02:06.879341 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:02:06.884462 19488 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43537 } }
I20260924 10:02:06.884820 19488 raft_consensus.cc:374] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:06.884946 19488 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30f3982063e49e382c3bf0eeae0e3ec, State: Initialized, Role: FOLLOWER
I20260924 10:02:06.885484 19488 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43537 } }
I20260924 10:02:06.885685 19488 raft_consensus.cc:388] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:06.885807 19488 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:06.885947 19488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:06.888077 19488 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43537 } }
I20260924 10:02:06.888540 19488 leader_election.cc:302] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e30f3982063e49e382c3bf0eeae0e3ec; no voters:
I20260924 10:02:06.889153 19488 leader_election.cc:288] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:06.889279 19493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:06.889889 19493 raft_consensus.cc:686] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [term 1 LEADER]: Becoming Leader. State: Replica: e30f3982063e49e382c3bf0eeae0e3ec, State: Running, Role: LEADER
I20260924 10:02:06.890404 19493 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43537 } }
I20260924 10:02:06.891142 19488 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:06.892901 19494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43537 } } }
I20260924 10:02:06.893329 19494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:06.893245 19495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30f3982063e49e382c3bf0eeae0e3ec. Latest consensus state: current_term: 1 leader_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3982063e49e382c3bf0eeae0e3ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 43537 } } }
I20260924 10:02:06.893509 19495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:06.894837 19501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:06.899679 19501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:06.908334 19501 catalog_manager.cc:1347] Generated new cluster ID: 63b33b7397d444e2b09a8b2ade24860e
I20260924 10:02:06.908433 19501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:06.929559 19501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:06.931001 19501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:06.942487 19501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e30f3982063e49e382c3bf0eeae0e3ec: Generated new TSK 0
I20260924 10:02:06.943459 19501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:02:07.018821 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:06.968005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.13.162.62:43537)
0924 10:02:06.968182 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:06.968188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:06.968546 (+ 358us) server_negotiation.cc:407] Connection header received
0924 10:02:06.968772 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:06.968776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:06.968856 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:06.968989 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:02:06.969421 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.970161 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.971540 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.972003 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.972122 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:06.972585 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:02:07.018104 (+ 45519us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:02:07.018508 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58032: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:02:07.018968 19491 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58032: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 10:02:07.019078 19517 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:06.967923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:43537 (local address 127.0.0.1:58032)
0924 10:02:06.968450 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:06.968464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:06.968595 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:06.969012 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:06.969017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:06.969029 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:02:06.969270 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:06.969279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.970307 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.970311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:06.971389 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:06.971401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:06.972169 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:06.972176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:06.972313 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:06.972345 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:02:07.018413 (+ 46068us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:02:07.018722 (+ 309us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.162.62:43537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
I20260924 10:02:07.020619 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19422
I20260924 10:02:07.021657 19483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:07.116707 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19422
2026-09-24T10:02:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (706 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (392 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (387 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:07 dist-test-slave-jvsj krb5kdc[19532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:07 dist-test-slave-jvsj krb5kdc[19532](info): set up 2 sockets
Sep 24 10:02:07 dist-test-slave-jvsj krb5kdc[19532](info): commencing operation
krb5kdc: starting...
W20260924 10:02:08.882699 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:09 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:02:09.069451 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42375
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:09.379803 19545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:09.380119 19545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:09.380234 19545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:09.387425 19545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:09.387548 19545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:09.387631 19545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:09.387707 19545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:09.387765 19545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:09.397568 19545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:42375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:09.398913 19545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:09.400197 19545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:09.411399 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:09.411458 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:09.411437 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:09.412199 19545 server_base.cc:1034] running on GCE node
I20260924 10:02:09.412923 19545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:09.414292 19545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:09.415531 19545 hybrid_clock.cc:648] HybridClock initialized: now 1790244129415472 us; error 50 us; skew 500 ppm
I20260924 10:02:09.415899 19545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:09 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244129, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:09.419281 19545 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:02:09.426677 19545 webserver.cc:460] Webserver started at http://127.13.162.62:36971/ using document root <none> and password file <none>
I20260924 10:02:09.427758 19545 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:09.428004 19545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:09.428438 19545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:09.432946 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "a877922c0390423c918f19cf4a57cd53"
format_stamp: "Formatted at 2026-09-24 10:02:09 on dist-test-slave-jvsj"
server_key: "3ae79c92080403856c4b4b67a310a4d7"
server_key_iv: "9104260e137e8e630aa1c13895f5c459"
server_key_version: "encryptionkey@0"
I20260924 10:02:09.434194 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "a877922c0390423c918f19cf4a57cd53"
format_stamp: "Formatted at 2026-09-24 10:02:09 on dist-test-slave-jvsj"
server_key: "3ae79c92080403856c4b4b67a310a4d7"
server_key_iv: "9104260e137e8e630aa1c13895f5c459"
server_key_version: "encryptionkey@0"
I20260924 10:02:09.443343 19545 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.001s
I20260924 10:02:09.447862 19560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:09.449608 19545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:02:09.449787 19545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "a877922c0390423c918f19cf4a57cd53"
format_stamp: "Formatted at 2026-09-24 10:02:09 on dist-test-slave-jvsj"
server_key: "3ae79c92080403856c4b4b67a310a4d7"
server_key_iv: "9104260e137e8e630aa1c13895f5c459"
server_key_version: "encryptionkey@0"
I20260924 10:02:09.449975 19545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:09.473112 19545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:09.477042 19545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:09.477303 19545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:09.493330 19545 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:42375
I20260924 10:02:09.493360 19611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:42375 every 8 connection(s)
I20260924 10:02:09.494735 19545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:09.497884 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19545
I20260924 10:02:09.498076 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:09.498878 13960 external_mini_cluster.cc:1428] Setting key 10cdb6b8222e29af4661614d893a8efd
I20260924 10:02:09.499423 19612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:02:09 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:09.515930 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53: Bootstrap starting.
I20260924 10:02:09.519254 19612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:09.520221 19612 log.cc:824] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:09.523314 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53: No bootstrap required, opened a new log
I20260924 10:02:09.523418 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:09.500871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.13.162.62:42375)
0924 10:02:09.501699 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:09.501715 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:09.501768 (+ 53us) server_negotiation.cc:407] Connection header received
0924 10:02:09.504208 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:09.504240 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:09.504784 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:09.506183 (+ 1399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:09.507036 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:09.509114 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:09.510832 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:09.511611 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:09.511771 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:09.516140 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:09.516153 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:09.516174 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:09.516218 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:09.519226 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:09.519909 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:09.519916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:09.519923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:09.520103 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:09.520455 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:09.520460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:09.520463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:09.521165 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:09.521461 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:09.521767 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 10:02:09.522109 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":577,"thread_start_us":349,"threads_started":1}
I20260924 10:02:09.531144 19612 raft_consensus.cc:348] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a877922c0390423c918f19cf4a57cd53" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42375 } }
I20260924 10:02:09.531507 19612 raft_consensus.cc:374] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:09.531595 19612 raft_consensus.cc:729] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a877922c0390423c918f19cf4a57cd53, State: Initialized, Role: FOLLOWER
I20260924 10:02:09.532126 19612 consensus_queue.cc:260] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a877922c0390423c918f19cf4a57cd53" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42375 } }
I20260924 10:02:09.532327 19612 raft_consensus.cc:388] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:09.532444 19612 raft_consensus.cc:482] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:09.532581 19612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:09.534795 19612 raft_consensus.cc:504] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a877922c0390423c918f19cf4a57cd53" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42375 } }
I20260924 10:02:09.535265 19612 leader_election.cc:302] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a877922c0390423c918f19cf4a57cd53; no voters:
I20260924 10:02:09.535794 19612 leader_election.cc:288] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:09.535941 19617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:09.536521 19617 raft_consensus.cc:686] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [term 1 LEADER]: Becoming Leader. State: Replica: a877922c0390423c918f19cf4a57cd53, State: Running, Role: LEADER
I20260924 10:02:09.537041 19617 consensus_queue.cc:237] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a877922c0390423c918f19cf4a57cd53" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42375 } }
I20260924 10:02:09.537720 19612 sys_catalog.cc:564] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:09.539592 19618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a877922c0390423c918f19cf4a57cd53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a877922c0390423c918f19cf4a57cd53" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42375 } } }
I20260924 10:02:09.539979 19618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:09.539601 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a877922c0390423c918f19cf4a57cd53. Latest consensus state: current_term: 1 leader_uuid: "a877922c0390423c918f19cf4a57cd53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a877922c0390423c918f19cf4a57cd53" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42375 } } }
I20260924 10:02:09.541561 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:09.542274 19623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:09.547489 19623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:09.555393 19623 catalog_manager.cc:1347] Generated new cluster ID: 704826ea9bd6480c87f468fe6b3ff9c0
I20260924 10:02:09.555506 19623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:09.563071 19623 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:09.564853 19623 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:09.572305 19623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a877922c0390423c918f19cf4a57cd53: Generated new TSK 0
I20260924 10:02:09.573302 19623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:02:09.625242 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42375
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:09.931404 19638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:09.931669 19638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:09.931799 19638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:09.939059 19638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:09.939193 19638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:09.939347 19638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:09.948859 19638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:09.950076 19638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:09.951391 19638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:09.963565 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:09.963562 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:09.963562 19646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:09.964365 19638 server_base.cc:1034] running on GCE node
I20260924 10:02:09.964946 19638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:09.965859 19638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:09.967056 19638 hybrid_clock.cc:648] HybridClock initialized: now 1790244129967001 us; error 42 us; skew 500 ppm
I20260924 10:02:09.967345 19638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:09 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244129, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:09.970155 19638 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:02:09.971453 19638 webserver.cc:460] Webserver started at http://127.13.162.1:44971/ using document root <none> and password file <none>
I20260924 10:02:09.972146 19638 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:09.972259 19638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:09.972529 19638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:09.975343 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "7644667569214cefac96b32069a2ce00"
format_stamp: "Formatted at 2026-09-24 10:02:09 on dist-test-slave-jvsj"
server_key: "4d8b7a5e41d4b8a7fd35a3c933fc07f3"
server_key_iv: "174e894450e70878a02c324fb1fddec7"
server_key_version: "encryptionkey@0"
I20260924 10:02:09.976053 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "7644667569214cefac96b32069a2ce00"
format_stamp: "Formatted at 2026-09-24 10:02:09 on dist-test-slave-jvsj"
server_key: "4d8b7a5e41d4b8a7fd35a3c933fc07f3"
server_key_iv: "174e894450e70878a02c324fb1fddec7"
server_key_version: "encryptionkey@0"
I20260924 10:02:09.981099 19638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:02:09.984638 19653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:09.985900 19638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:09.986089 19638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "7644667569214cefac96b32069a2ce00"
format_stamp: "Formatted at 2026-09-24 10:02:09 on dist-test-slave-jvsj"
server_key: "4d8b7a5e41d4b8a7fd35a3c933fc07f3"
server_key_iv: "174e894450e70878a02c324fb1fddec7"
server_key_version: "encryptionkey@0"
I20260924 10:02:09.986270 19638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:09.998116 19638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:10.001892 19638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:10.002193 19638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:10.003218 19638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:10.004721 19638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:10.004789 19638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.004874 19638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:10.004916 19638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.032929 19638 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:43937
I20260924 10:02:10.032944 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:43937 every 8 connection(s)
I20260924 10:02:10.034134 19638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:10.043438 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19638
I20260924 10:02:10.043654 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:10.044275 13960 external_mini_cluster.cc:1428] Setting key 67a150746bfe928dd71f89e319d62dd9
Sep 24 10:02:10 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244129, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:10.052393 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:10.036527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:46309 (local address 127.13.162.62:42375)
0924 10:02:10.036983 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:10.036990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:10.037823 (+ 833us) server_negotiation.cc:407] Connection header received
0924 10:02:10.039715 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:10.039721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:10.039801 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:10.039897 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:10.041309 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.042067 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.043569 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.044042 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.044149 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:10.047278 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:10.047284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:10.047288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:10.047316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:10.049652 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:10.050300 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:10.050304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:10.050306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:10.050444 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:10.050828 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:10.050833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:10.050837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:10.051082 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:10.051218 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:10.051842 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 10:02:10.052053 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260924 10:02:10.053313 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:10.037004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42375 (local address 127.13.162.1:46309)
0924 10:02:10.037680 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:10.037717 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:10.039461 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:10.040133 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:10.040141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:10.040609 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:10.041156 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:10.041173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.042206 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.042210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:10.043444 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:10.043454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.044251 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.044258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:10.044488 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:10.045062 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:10.045082 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:10.047081 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:10.049772 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:10.049781 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:10.049798 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:10.050188 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:10.050537 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:10.050543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:10.050547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:10.050719 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:10.051195 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:10.051203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:10.051519 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 10:02:10.051736 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 10:02:10.052246 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":253,"threads_started":1}
I20260924 10:02:10.054708 19766 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42375
I20260924 10:02:10.054971 19766 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:10.055686 19766 heartbeater.cc:511] Master 127.13.162.62:42375 requested a full tablet report, sending...
I20260924 10:02:10.057564 19577 ts_manager.cc:194] Registered new tserver with Master: 7644667569214cefac96b32069a2ce00 (127.13.162.1:43937)
I20260924 10:02:10.058974 19577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:46309
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:02:10.083321 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42375
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:10.391925 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:10.392205 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:10.392333 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:10.400317 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:10.400429 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:10.400579 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:10.410226 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:10.411535 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:10.412825 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:10.425350 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:10.425314 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:10.425318 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:10.426242 19772 server_base.cc:1034] running on GCE node
I20260924 10:02:10.426890 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:10.427939 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:10.429149 19772 hybrid_clock.cc:648] HybridClock initialized: now 1790244130429081 us; error 61 us; skew 500 ppm
I20260924 10:02:10.429464 19772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:10 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244130, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:10.432384 19772 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:02:10.433713 19772 webserver.cc:460] Webserver started at http://127.13.162.2:33733/ using document root <none> and password file <none>
I20260924 10:02:10.434424 19772 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:10.434525 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:10.434782 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:10.437649 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "a607db088dfe49f68b5c02372ed37095"
format_stamp: "Formatted at 2026-09-24 10:02:10 on dist-test-slave-jvsj"
server_key: "dad1b785f1a5a9be15a5b7d862c7f2eb"
server_key_iv: "c5ea4714b4fe91dbcca7b81047efe14b"
server_key_version: "encryptionkey@0"
I20260924 10:02:10.438477 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "a607db088dfe49f68b5c02372ed37095"
format_stamp: "Formatted at 2026-09-24 10:02:10 on dist-test-slave-jvsj"
server_key: "dad1b785f1a5a9be15a5b7d862c7f2eb"
server_key_iv: "c5ea4714b4fe91dbcca7b81047efe14b"
server_key_version: "encryptionkey@0"
I20260924 10:02:10.443565 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:02:10.446768 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.447996 19772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:02:10.448122 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "a607db088dfe49f68b5c02372ed37095"
format_stamp: "Formatted at 2026-09-24 10:02:10 on dist-test-slave-jvsj"
server_key: "dad1b785f1a5a9be15a5b7d862c7f2eb"
server_key_iv: "c5ea4714b4fe91dbcca7b81047efe14b"
server_key_version: "encryptionkey@0"
I20260924 10:02:10.448292 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:10.472342 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:10.476595 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:10.476862 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:10.477818 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:10.479125 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:10.479203 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.479318 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:10.479368 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.508319 19772 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:44019
I20260924 10:02:10.508342 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:44019 every 8 connection(s)
I20260924 10:02:10.509867 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:10.510967 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19772
I20260924 10:02:10.511161 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:10.511663 13960 external_mini_cluster.cc:1428] Setting key f0fb9dafdb8f83943f8f9df248edd8c1
Sep 24 10:02:10 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244130, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:02:10.530634 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:10.513288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:34041 (local address 127.13.162.62:42375)
0924 10:02:10.513440 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:10.513446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:10.514173 (+ 727us) server_negotiation.cc:407] Connection header received
0924 10:02:10.516739 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:10.516745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:10.516837 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:10.516937 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:10.518402 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.519218 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.520875 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.521510 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.521659 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:10.525172 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:10.525178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:10.525181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:10.525212 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:10.527662 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:10.528534 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:10.528538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:10.528540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:10.528665 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:10.529103 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:10.529106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:10.529108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:10.529388 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:10.529503 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:10.530217 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 10:02:10.530410 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:02:10.531777 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:10.513200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42375 (local address 127.13.162.2:34041)
0924 10:02:10.514016 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:10.514070 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:10.516399 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:10.517143 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:10.517151 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:10.517664 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:10.518225 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:10.518242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.519383 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.519388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:10.520736 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:10.520748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:10.521734 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:10.521739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:10.521931 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:10.522590 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:10.522611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:10.524924 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:10.527805 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:10.527815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:10.527834 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:10.528415 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:10.528770 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:10.528775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:10.528779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:10.529011 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:10.529505 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:10.529512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:10.529843 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 10:02:10.530108 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 10:02:10.530557 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260924 10:02:10.533313 19900 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42375
I20260924 10:02:10.533620 19900 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:10.534286 19900 heartbeater.cc:511] Master 127.13.162.62:42375 requested a full tablet report, sending...
I20260924 10:02:10.535741 19577 ts_manager.cc:194] Registered new tserver with Master: a607db088dfe49f68b5c02372ed37095 (127.13.162.2:44019)
I20260924 10:02:10.536516 19577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:34041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:02:10.553500 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42375
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:10.863173 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:10.863452 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:10.863586 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:10.870802 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:10.870914 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:10.871070 19905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:10.880722 19905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:10.881966 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:10.883257 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:10.895514 19910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:10.895589 19911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:10.895565 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:10.896812 19905 server_base.cc:1034] running on GCE node
I20260924 10:02:10.897406 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:10.898334 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:10.899535 19905 hybrid_clock.cc:648] HybridClock initialized: now 1790244130899490 us; error 48 us; skew 500 ppm
I20260924 10:02:10.899860 19905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:10 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244130, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:10.902849 19905 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:02:10.904253 19905 webserver.cc:460] Webserver started at http://127.13.162.3:45969/ using document root <none> and password file <none>
I20260924 10:02:10.904942 19905 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:10.905035 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:10.905315 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:10.908165 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "231bdb7be33942fb81d2205c934614ea"
format_stamp: "Formatted at 2026-09-24 10:02:10 on dist-test-slave-jvsj"
server_key: "eb7610736645c490646c8f7563620425"
server_key_iv: "38703107837c4d3ac8b6b82eaf0ecd15"
server_key_version: "encryptionkey@0"
I20260924 10:02:10.908903 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "231bdb7be33942fb81d2205c934614ea"
format_stamp: "Formatted at 2026-09-24 10:02:10 on dist-test-slave-jvsj"
server_key: "eb7610736645c490646c8f7563620425"
server_key_iv: "38703107837c4d3ac8b6b82eaf0ecd15"
server_key_version: "encryptionkey@0"
I20260924 10:02:10.913988 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:02:10.917486 19920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.918980 19905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:10.919123 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "231bdb7be33942fb81d2205c934614ea"
format_stamp: "Formatted at 2026-09-24 10:02:10 on dist-test-slave-jvsj"
server_key: "eb7610736645c490646c8f7563620425"
server_key_iv: "38703107837c4d3ac8b6b82eaf0ecd15"
server_key_version: "encryptionkey@0"
I20260924 10:02:10.919312 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:10.960784 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:10.964789 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:10.965081 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:10.966006 19905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:10.967340 19905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:10.967408 19905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.967500 19905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:10.967545 19905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:10.996145 19905 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:46059
I20260924 10:02:10.996191 20032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:46059 every 8 connection(s)
I20260924 10:02:10.997362 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:11.002346 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 19905
I20260924 10:02:11.002514 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:11.003077 13960 external_mini_cluster.cc:1428] Setting key c15c3a594c6feeba4e46a55f49482e0f
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244130, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:11.016472 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:10.999738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:42785 (local address 127.13.162.62:42375)
0924 10:02:10.999925 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:10.999931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.001087 (+ 1156us) server_negotiation.cc:407] Connection header received
0924 10:02:11.003322 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.003327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.003396 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.003482 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.005274 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.006093 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.007683 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.008060 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.008155 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.011367 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.011372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.011375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.011407 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.013801 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.014469 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.014473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.014476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.014584 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.015024 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.015027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.015029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.015308 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.015410 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.016067 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.016229 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:02:11.017812 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.000245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42375 (local address 127.13.162.3:42785)
0924 10:02:11.000934 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:11.000971 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:11.003017 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:11.003759 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:11.003768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:11.004388 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:11.005111 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:11.005136 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.006247 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.006250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:11.007558 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:11.007575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.008263 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.008268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:11.008468 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.009110 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:11.009131 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:11.011186 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:11.013922 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:11.013929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:11.013943 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:11.014374 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:11.014714 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:11.014718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:11.014720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:11.014913 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:11.015454 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:11.015461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:11.015777 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 10:02:11.016005 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 10:02:11.016477 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":298,"threads_started":1}
I20260924 10:02:11.019105 20033 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42375
I20260924 10:02:11.019361 20033 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:11.020001 20033 heartbeater.cc:511] Master 127.13.162.62:42375 requested a full tablet report, sending...
I20260924 10:02:11.021266 19577 ts_manager.cc:194] Registered new tserver with Master: 231bdb7be33942fb81d2205c934614ea (127.13.162.3:46059)
I20260924 10:02:11.022045 19577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:42785
I20260924 10:02:11.030444 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/test-admin.keytab.
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.055454 13960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 10:02:11.060247 20040 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:11.057916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.13.162.1:43937)
0924 10:02:11.058594 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.058602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.058638 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:02:11.058700 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.058707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.058925 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.059052 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.059869 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53698: BlockingRecv error: recv got EOF from 127.0.0.1:53698 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":382,"threads_started":1}
W20260924 10:02:11.060397 13960 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.061750 19766 heartbeater.cc:503] Master 127.13.162.62:42375 was elected leader, sending a full tablet report...
I20260924 10:02:11.062049 13960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.063575 13960 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:02:11.077486 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.065163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.13.162.1:43937)
0924 10:02:11.065338 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.065343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.065503 (+ 160us) server_negotiation.cc:407] Connection header received
0924 10:02:11.065653 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.065657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.065713 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.065788 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.066690 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.067651 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.068800 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.069355 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.069444 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.072723 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.072734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.072757 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.072789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.075228 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.075820 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.075828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.075830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.075938 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.076208 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.076211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.076213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.076528 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.076741 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.076976 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.077210 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:02:11.082551 20040 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:11.081211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.13.162.1:43937)
0924 10:02:11.081393 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.081398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.081522 (+ 124us) server_negotiation.cc:407] Connection header received
0924 10:02:11.081648 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.081652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.081712 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.081779 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.082360 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53708: BlockingRecv error: recv got EOF from 127.0.0.1:53708 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:02:11.082683 13960 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.084038 13960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.085129 13960 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:02:11.097785 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.086279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.13.162.1:43937)
0924 10:02:11.086475 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.086481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.086728 (+ 247us) server_negotiation.cc:407] Connection header received
0924 10:02:11.086857 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.086862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.086920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.086991 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.087809 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.088482 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.089619 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.090230 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.090380 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.093466 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.093470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.093473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.093499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.095726 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.096303 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.096307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.096309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.096410 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.096735 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.096737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.096739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.096957 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.097164 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.097451 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.097586 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 10:02:11.101533 20040 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:11.100067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.13.162.1:43937)
0924 10:02:11.100225 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.100230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.100406 (+ 176us) server_negotiation.cc:407] Connection header received
0924 10:02:11.100542 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.100546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.100607 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.100699 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.101329 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53724: BlockingRecv error: recv got EOF from 127.0.0.1:53724 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:02:11.101729 13960 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244085964582-13960-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.103196 13960 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:11.104349 13960 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:02:11.116837 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.105500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.13.162.1:43937)
0924 10:02:11.105688 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.105693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.105891 (+ 198us) server_negotiation.cc:407] Connection header received
0924 10:02:11.106013 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.106017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.106067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.106127 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.107007 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.107690 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.108808 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.109170 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.109248 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.112435 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.112439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.112443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.112470 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.114847 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.115383 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.115386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.115389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.115516 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.115775 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.115778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.115780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.116016 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.116204 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.116377 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.116518 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:02:11.118194 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19638
I20260924 10:02:11.133862 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.118831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.13.162.1:43937)
0924 10:02:11.118949 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.118953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.119293 (+ 340us) server_negotiation.cc:407] Connection header received
0924 10:02:11.119397 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.119400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.119448 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.119514 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.120312 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.120946 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.122184 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.122648 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.122730 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.129571 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.129576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.129580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.129609 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.131850 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.132397 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.132400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.132402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.132516 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.132779 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.132782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.132784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.133012 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.133189 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.133462 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.133610 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:02:11.134486 19761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:11.214984 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19638
I20260924 10:02:11.227891 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19772
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:02:11.242086 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.228749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.13.162.2:44019)
0924 10:02:11.229275 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.229281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.229307 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:02:11.229365 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.229372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.229578 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.229706 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.230626 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.231648 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.232798 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.233350 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.233448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.236877 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.236887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.236902 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.236942 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.239687 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.240318 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.240325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.240327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.240437 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.240733 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.240736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.240738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.241084 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.241349 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.241606 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.241802 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":246,"threads_started":1}
I20260924 10:02:11.243602 19895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:11.323211 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19772
I20260924 10:02:11.335391 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19905
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:02:11.350669 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:11.336336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.13.162.3:46059)
0924 10:02:11.336963 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:11.336970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:11.337011 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:02:11.337068 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:11.337074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:11.337376 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:11.337532 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:11.338645 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.339730 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.340931 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:11.341517 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:11.341632 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:11.345509 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:11.345520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:11.345535 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:11.345573 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:11.348295 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.348868 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.348875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.348877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.349040 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:11.349318 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:11.349321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:11.349323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:11.349655 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:11.349881 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:11.350084 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 10:02:11.350278 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":334,"threads_started":1}
I20260924 10:02:11.351867 20028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:11.434046 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19905
I20260924 10:02:11.446343 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 19545
I20260924 10:02:11.447466 19607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:11.546185 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 19545
2026-09-24T10:02:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3654 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[20060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[20060](info): set up 2 sockets
Sep 24 10:02:11 dist-test-slave-jvsj krb5kdc[20060](info): commencing operation
krb5kdc: starting...
W20260924 10:02:12.546770 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:12 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab.
Entry for principal oryx/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab.
I20260924 10:02:12.737077 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44465
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab
--principal=oryx/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:13.045383 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:13.045713 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:13.045847 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:13.052968 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:13.053092 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:13.053170 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:13.053246 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:13.053303 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:13.063143 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab
--principal=oryx/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:13.064498 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:13.065780 20073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:13.076989 20079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:13.077105 20078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:13.077028 20081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:13.078006 20073 server_base.cc:1034] running on GCE node
I20260924 10:02:13.078810 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:13.080185 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:13.081418 20073 hybrid_clock.cc:648] HybridClock initialized: now 1790244133081359 us; error 42 us; skew 500 ppm
I20260924 10:02:13.081768 20073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:13 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244133, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:13.084764 20073 init.cc:377] Logged in from keytab as oryx/127.13.162.62@KRBTEST.COM (short username oryx)
I20260924 10:02:13.086102 20073 webserver.cc:460] Webserver started at http://127.13.162.62:36085/ using document root <none> and password file <none>
I20260924 10:02:13.086767 20073 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:13.086863 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:13.087110 20073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:13.089881 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "643687b7468c438ebc39c28da7f05a82"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "694de5c4a97707ff72f89653dd676d25"
server_key_iv: "aaaac4d9f5c079b97153ab7208aa5939"
server_key_version: "encryptionkey@0"
I20260924 10:02:13.090759 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "643687b7468c438ebc39c28da7f05a82"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "694de5c4a97707ff72f89653dd676d25"
server_key_iv: "aaaac4d9f5c079b97153ab7208aa5939"
server_key_version: "encryptionkey@0"
I20260924 10:02:13.095683 20073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 10:02:13.098922 20088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:13.100226 20073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:02:13.100374 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "643687b7468c438ebc39c28da7f05a82"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "694de5c4a97707ff72f89653dd676d25"
server_key_iv: "aaaac4d9f5c079b97153ab7208aa5939"
server_key_version: "encryptionkey@0"
I20260924 10:02:13.100550 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:13.119580 20073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:13.123562 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:13.123816 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:13.140260 20073 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:44465
I20260924 10:02:13.140281 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:44465 every 8 connection(s)
I20260924 10:02:13.141595 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:13.144977 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20073
I20260924 10:02:13.145160 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:13.145385 20140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:13.145865 13960 external_mini_cluster.cc:1428] Setting key 4367cfee835d2dd558d2bc79f74d470f
I20260924 10:02:13.159307 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Bootstrap starting.
Sep 24 10:02:13 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:13.162917 20140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:13.164248 20140 log.cc:824] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:13.167694 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: No bootstrap required, opened a new log
I20260924 10:02:13.168926 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:13.147845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.13.162.62:44465)
0924 10:02:13.148659 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:13.148675 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:13.148727 (+ 52us) server_negotiation.cc:407] Connection header received
0924 10:02:13.150470 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:13.150506 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:13.151024 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:13.151484 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:13.152717 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:13.154618 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:13.155876 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:13.156940 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:13.157138 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:13.160855 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:13.160866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:13.160892 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:13.160947 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:13.164567 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:13.165364 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:13.165371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:13.165377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:13.165541 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:13.165910 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:13.165915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:13.165918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:13.166595 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:13.166863 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:13.167200 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 10:02:13.167616 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":337,"threads_started":1}
I20260924 10:02:13.175527 20140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } }
I20260924 10:02:13.175917 20140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:13.176018 20140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643687b7468c438ebc39c28da7f05a82, State: Initialized, Role: FOLLOWER
I20260924 10:02:13.176632 20140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } }
I20260924 10:02:13.176842 20140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:13.176967 20140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:13.177120 20140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:13.179255 20140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } }
I20260924 10:02:13.179739 20140 leader_election.cc:302] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643687b7468c438ebc39c28da7f05a82; no voters:
I20260924 10:02:13.180338 20140 leader_election.cc:288] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:13.180445 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:13.181030 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 LEADER]: Becoming Leader. State: Replica: 643687b7468c438ebc39c28da7f05a82, State: Running, Role: LEADER
I20260924 10:02:13.181627 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } }
I20260924 10:02:13.182569 20140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:13.184491 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } }
I20260924 10:02:13.184546 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643687b7468c438ebc39c28da7f05a82. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } }
I20260924 10:02:13.184863 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:13.184901 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:13.193938 20152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:13.199038 20152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:13.207134 20152 catalog_manager.cc:1347] Generated new cluster ID: e2e4455f26a84d5aabfb9cb777021f6b
I20260924 10:02:13.207259 20152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:13.226219 20152 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:13.227535 20152 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:13.234683 20152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Generated new TSK 0
I20260924 10:02:13.235790 20152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab.
Entry for principal oryx/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab.
I20260924 10:02:13.291201 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44465
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:13.607512 20166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:13.607789 20166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:13.607905 20166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:13.615270 20166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:13.615387 20166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:13.615531 20166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:13.625392 20166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:13.626791 20166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:13.628125 20166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:13.640859 20171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:13.640823 20172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:13.640894 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:13.641690 20166 server_base.cc:1034] running on GCE node
I20260924 10:02:13.642285 20166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:13.643258 20166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:13.644481 20166 hybrid_clock.cc:648] HybridClock initialized: now 1790244133644416 us; error 55 us; skew 500 ppm
I20260924 10:02:13.644815 20166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:13 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244133, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:13.647996 20166 init.cc:377] Logged in from keytab as oryx/127.13.162.1@KRBTEST.COM (short username oryx)
I20260924 10:02:13.649376 20166 webserver.cc:460] Webserver started at http://127.13.162.1:42423/ using document root <none> and password file <none>
I20260924 10:02:13.650017 20166 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:13.650115 20166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:13.650403 20166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:13.653259 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "9fa49ed9187a42568c7ff491e12fb40b"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "eeb7501a4c3073da9aae28760bb61404"
server_key_iv: "ad2d6cdfff08a147fed2ca0e72804f27"
server_key_version: "encryptionkey@0"
I20260924 10:02:13.653993 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "9fa49ed9187a42568c7ff491e12fb40b"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "eeb7501a4c3073da9aae28760bb61404"
server_key_iv: "ad2d6cdfff08a147fed2ca0e72804f27"
server_key_version: "encryptionkey@0"
I20260924 10:02:13.659170 20166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 10:02:13.662598 20181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:13.664057 20166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:13.664211 20166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "9fa49ed9187a42568c7ff491e12fb40b"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "eeb7501a4c3073da9aae28760bb61404"
server_key_iv: "ad2d6cdfff08a147fed2ca0e72804f27"
server_key_version: "encryptionkey@0"
I20260924 10:02:13.664391 20166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:13.688783 20166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:13.692991 20166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:13.693249 20166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:13.694198 20166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:13.695502 20166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:13.695582 20166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:13.695704 20166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:13.695752 20166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:13.725193 20166 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:45073
I20260924 10:02:13.725237 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:45073 every 8 connection(s)
I20260924 10:02:13.726480 20166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:13.729468 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20166
I20260924 10:02:13.729694 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:13.730739 13960 external_mini_cluster.cc:1428] Setting key c49d7a30661a59f0b084025c219c3e2e
Sep 24 10:02:13 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244133, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.2@KRBTEST.COM" created.
I20260924 10:02:13.762279 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:13.729636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:36149)
0924 10:02:13.741353 (+ 11717us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:13.741420 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:13.743919 (+ 2499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:13.745018 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:13.745030 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:13.745778 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:13.746623 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:13.746650 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:13.748091 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:13.748097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:13.750200 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:13.750215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:13.751281 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:13.751290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:13.751611 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:13.752411 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:13.752442 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:13.754792 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:13.758173 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:13.758182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:13.758202 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:13.758821 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:13.759233 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:13.759238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:13.759243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:13.759420 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:13.759986 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:13.759995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:13.760390 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 10:02:13.760635 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:02:13.761096 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11334,"thread_start_us":11609,"threads_started":1}
I20260924 10:02:13.763056 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:13.729110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:36149 (local address 127.13.162.62:44465)
0924 10:02:13.732444 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:13.732450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:13.742371 (+ 9921us) server_negotiation.cc:407] Connection header received
0924 10:02:13.744491 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:13.744497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:13.744587 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:13.744695 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:13.746852 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:13.747836 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:13.750380 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:13.750975 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:13.751127 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:13.755114 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:13.755121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:13.755126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:13.755167 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:13.758045 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:13.758956 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:13.758962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:13.758965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:13.759124 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:13.759522 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:13.759527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:13.759531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:13.759851 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:13.759986 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:13.762499 (+ 2513us) server_negotiation.cc:299] Negotiation successful
0924 10:02:13.762721 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3218,"thread_start_us":3114,"threads_started":1}
I20260924 10:02:13.764209 20294 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44465
I20260924 10:02:13.764541 20294 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:13.765403 20294 heartbeater.cc:511] Master 127.13.162.62:44465 requested a full tablet report, sending...
I20260924 10:02:13.767884 20105 ts_manager.cc:194] Registered new tserver with Master: 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073)
I20260924 10:02:13.769723 20105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:36149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab.
Entry for principal oryx/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab.
I20260924 10:02:13.785594 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44465
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:14.087924 20300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:14.088197 20300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:14.088292 20300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:14.095189 20300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:14.095283 20300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:14.095394 20300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:14.104777 20300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:14.106026 20300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:14.107272 20300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:14.119465 20305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:14.119464 20306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:14.119640 20308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:14.120199 20300 server_base.cc:1034] running on GCE node
I20260924 10:02:14.120769 20300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:14.121685 20300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:14.122907 20300 hybrid_clock.cc:648] HybridClock initialized: now 1790244134122857 us; error 31 us; skew 500 ppm
I20260924 10:02:14.123234 20300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:14 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244134, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:14.126374 20300 init.cc:377] Logged in from keytab as oryx/127.13.162.2@KRBTEST.COM (short username oryx)
I20260924 10:02:14.127765 20300 webserver.cc:460] Webserver started at http://127.13.162.2:42117/ using document root <none> and password file <none>
I20260924 10:02:14.128419 20300 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:14.128539 20300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:14.128813 20300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:14.131726 20300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "82add254d9d8479480e7018a651d2291"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "339e14692eacb2d825436c3e9edc91b5"
server_key_iv: "3be672e9f6a8ed5f23848cc1f8ad15ea"
server_key_version: "encryptionkey@0"
I20260924 10:02:14.132469 20300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "82add254d9d8479480e7018a651d2291"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "339e14692eacb2d825436c3e9edc91b5"
server_key_iv: "3be672e9f6a8ed5f23848cc1f8ad15ea"
server_key_version: "encryptionkey@0"
I20260924 10:02:14.137811 20300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 10:02:14.141422 20315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:14.142877 20300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:14.143020 20300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "82add254d9d8479480e7018a651d2291"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "339e14692eacb2d825436c3e9edc91b5"
server_key_iv: "3be672e9f6a8ed5f23848cc1f8ad15ea"
server_key_version: "encryptionkey@0"
I20260924 10:02:14.143219 20300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:14.174103 20300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:14.178180 20300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:14.178503 20300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:14.179435 20300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:14.180684 20300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:14.180756 20300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:14.180871 20300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:14.180927 20300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:14.209528 20300 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:44257
I20260924 10:02:14.209569 20427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:44257 every 8 connection(s)
I20260924 10:02:14.210784 20300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:14.212917 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20300
I20260924 10:02:14.213094 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:14.213584 13960 external_mini_cluster.cc:1428] Setting key 19b43e43048698f20f694614b4f6bb9f
Sep 24 10:02:14 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244134, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:14.229768 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:14.213258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:37497 (local address 127.13.162.62:44465)
0924 10:02:14.213450 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:14.213456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:14.214645 (+ 1189us) server_negotiation.cc:407] Connection header received
0924 10:02:14.216427 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:14.216432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:14.216510 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:14.216594 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:14.218093 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.218848 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.220361 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.220764 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.220872 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:14.224301 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:14.224306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:14.224309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:14.224340 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:14.226941 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:14.227612 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:14.227618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:14.227621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:14.227749 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:14.228152 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:14.228156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:14.228158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:14.228411 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:14.228535 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:14.229208 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 10:02:14.229432 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for oryx/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:02:14.230747 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:14.213715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:37497)
0924 10:02:14.214484 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:14.214530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:14.216161 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:14.216844 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:14.216854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:14.217373 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:14.217932 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:14.217949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.218994 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.218998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:14.220226 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:14.220238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.220972 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.220979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:14.221214 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:14.221864 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:14.221885 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:14.224091 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:14.227073 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:14.227082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:14.227098 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:14.227500 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:14.227850 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:14.227855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:14.227859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:14.228042 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:14.228526 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:14.228533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:14.228861 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 10:02:14.229098 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 10:02:14.229619 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":301,"threads_started":1}
I20260924 10:02:14.232146 20428 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44465
I20260924 10:02:14.232425 20428 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.3@KRBTEST.COM" created.
I20260924 10:02:14.233237 20428 heartbeater.cc:511] Master 127.13.162.62:44465 requested a full tablet report, sending...
I20260924 10:02:14.234853 20105 ts_manager.cc:194] Registered new tserver with Master: 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257)
I20260924 10:02:14.235745 20105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:37497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab.
Entry for principal oryx/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab.
I20260924 10:02:14.255561 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44465
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:14.573163 20433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:14.573441 20433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:14.573565 20433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:14.580760 20433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:14.580878 20433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:14.581024 20433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:14.590785 20433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:14.592012 20433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:14.593254 20433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:14.605388 20439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:14.605389 20441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:14.605598 20438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:14.606215 20433 server_base.cc:1034] running on GCE node
I20260924 10:02:14.606856 20433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:14.607779 20433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:14.608974 20433 hybrid_clock.cc:648] HybridClock initialized: now 1790244134608925 us; error 46 us; skew 500 ppm
I20260924 10:02:14.609283 20433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:14 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244134, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:14.612336 20433 init.cc:377] Logged in from keytab as oryx/127.13.162.3@KRBTEST.COM (short username oryx)
I20260924 10:02:14.613854 20433 webserver.cc:460] Webserver started at http://127.13.162.3:41135/ using document root <none> and password file <none>
I20260924 10:02:14.614537 20433 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:14.614636 20433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:14.614880 20433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:14.617682 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "f86c2f01f67b41d09e1f94f36c9872cd"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "ecd1ef9fc0179c8a1c248e98e9580f1b"
server_key_iv: "9a6f797304b7054c7a56528b92795178"
server_key_version: "encryptionkey@0"
I20260924 10:02:14.618423 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "f86c2f01f67b41d09e1f94f36c9872cd"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "ecd1ef9fc0179c8a1c248e98e9580f1b"
server_key_iv: "9a6f797304b7054c7a56528b92795178"
server_key_version: "encryptionkey@0"
I20260924 10:02:14.623247 20433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:02:14.626601 20448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:14.628015 20433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:14.628177 20433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "f86c2f01f67b41d09e1f94f36c9872cd"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "ecd1ef9fc0179c8a1c248e98e9580f1b"
server_key_iv: "9a6f797304b7054c7a56528b92795178"
server_key_version: "encryptionkey@0"
I20260924 10:02:14.628441 20433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:14.648289 20433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:14.652426 20433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:14.652665 20433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:14.653590 20433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:14.654884 20433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:14.654955 20433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:14.655043 20433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:14.655084 20433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:14.684098 20433 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:33439
I20260924 10:02:14.684141 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:33439 every 8 connection(s)
I20260924 10:02:14.685310 20433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:14.693212 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20433
I20260924 10:02:14.693369 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:14.693929 13960 external_mini_cluster.cc:1428] Setting key c6fbc5b5ea3db6a0360ea4b2c3722531
Sep 24 10:02:14 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244134, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:14.704289 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:14.687812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:51021 (local address 127.13.162.62:44465)
0924 10:02:14.687964 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:14.687969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:14.689152 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 10:02:14.690927 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:14.690932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:14.691028 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:14.691132 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:14.692615 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.693402 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.695220 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.695659 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.695770 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:14.699153 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:14.699159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:14.699163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:14.699197 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:14.701429 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:14.702105 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:14.702110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:14.702113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:14.702229 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:14.702655 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:14.702658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:14.702660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:14.702925 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:14.703051 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:14.703735 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 10:02:14.703942 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:02:14.705237 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:14.688281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:51021)
0924 10:02:14.688991 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:14.689032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:14.690661 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:14.691386 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:14.691394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:14.691874 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:14.692434 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:14.692451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.693553 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.693558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:14.695103 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:14.695124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:14.695869 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:14.695876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:14.696088 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:14.696762 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:14.696788 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:14.698948 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:14.701557 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:14.701566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:14.701584 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:14.701983 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:14.702353 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:14.702358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:14.702362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:14.702538 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:14.703043 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:14.703051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:14.703385 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 10:02:14.703615 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 10:02:14.704151 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":292,"threads_started":1}
I20260924 10:02:14.706677 20561 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44465
I20260924 10:02:14.706964 20561 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:14.707792 20561 heartbeater.cc:511] Master 127.13.162.62:44465 requested a full tablet report, sending...
I20260924 10:02:14.709158 20104 ts_manager.cc:194] Registered new tserver with Master: f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439)
I20260924 10:02:14.709975 20104 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:51021
I20260924 10:02:14.721150 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.162.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.61.keytab.
Entry for principal oryx/127.13.162.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.61.keytab.
I20260924 10:02:14.765081 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:45497
--webserver_interface=127.13.162.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.61.keytab
--principal=oryx/127.13.162.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44465,127.13.162.61:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:02:14.773521 20294 heartbeater.cc:503] Master 127.13.162.62:44465 was elected leader, sending a full tablet report...
W20260924 10:02:15.075536 20566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:15.075850 20566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:15.075976 20566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:15.083379 20566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:15.083503 20566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:15.083575 20566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:15.083647 20566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:15.083702 20566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:15.093562 20566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44465,127.13.162.61:45497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.61.keytab
--principal=oryx/127.13.162.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.162.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
W20260924 10:02:15.093919 20566 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.162.62:44465,127.13.162.61:45497'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 10:02:15.094970 20566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:15.096225 20566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:15.106958 20571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:15.107030 20572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:15.106999 20574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:15.107757 20566 server_base.cc:1034] running on GCE node
I20260924 10:02:15.108495 20566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:15.109787 20566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:15.110983 20566 hybrid_clock.cc:648] HybridClock initialized: now 1790244135110913 us; error 54 us; skew 500 ppm
I20260924 10:02:15.111279 20566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:15.114185 20566 init.cc:377] Logged in from keytab as oryx/127.13.162.61@KRBTEST.COM (short username oryx)
I20260924 10:02:15.115588 20566 webserver.cc:460] Webserver started at http://127.13.162.61:40923/ using document root <none> and password file <none>
I20260924 10:02:15.116241 20566 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:15.116334 20566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:15.116617 20566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:15.119475 20566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/instance:
uuid: "4b7af120325d4809b3aff9cd14e1026c"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "b15657ed675af3cda80a3e3ba29fe7b9"
server_key_iv: "c0d5a608325da2cafeff8de022079ce3"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.120213 20566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal/instance:
uuid: "4b7af120325d4809b3aff9cd14e1026c"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "b15657ed675af3cda80a3e3ba29fe7b9"
server_key_iv: "c0d5a608325da2cafeff8de022079ce3"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.125197 20566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:02:15.128660 20581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:15.130188 20566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:15.130374 20566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
uuid: "4b7af120325d4809b3aff9cd14e1026c"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "b15657ed675af3cda80a3e3ba29fe7b9"
server_key_iv: "c0d5a608325da2cafeff8de022079ce3"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.130578 20566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:15.152786 20566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:15.156764 20566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:15.157003 20566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:15.173949 20566 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.61:45497
I20260924 10:02:15.173956 20632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.61:45497 every 8 connection(s)
I20260924 10:02:15.175297 20566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
I20260924 10:02:15.178980 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:15.184019 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20566
I20260924 10:02:15.185940 20633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.61@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
I20260924 10:02:15.203977 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.188785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.13.162.62:44465)
0924 10:02:15.188928 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.188932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.189630 (+ 698us) server_negotiation.cc:407] Connection header received
0924 10:02:15.189815 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.189820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.189899 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.189982 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.191455 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.192261 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.193848 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.194254 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.194386 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.197795 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.197800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.197804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.197838 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.200514 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.201436 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.201440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.201443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.201576 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.201984 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.201989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.201993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.202449 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.202570 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.203426 (+ 856us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.203641 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:02:15.205448 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.185979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.13.162.61:45497)
0924 10:02:15.186765 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.186777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.186819 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:02:15.188841 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.188874 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.189438 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.190449 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.191308 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.192586 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.193875 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.194445 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.194604 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.198635 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.198643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.198656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.198689 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.201602 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.202373 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.202379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.202381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.202489 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.202856 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.202860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.202863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.203257 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.203590 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.203816 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.204077 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"spinlock_wait_cycles":30976,"thread_start_us":391,"threads_started":1}
I20260924 10:02:15.205488 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.188808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37224)
0924 10:02:15.189463 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.189497 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.189690 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.190061 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.190072 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.190888 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.191266 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.191290 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.192415 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.192420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.193705 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.193716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.194495 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.194503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.194648 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.195285 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.195306 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.197563 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.200646 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.200655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.200668 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.201306 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.201684 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.201689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.201692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.201880 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.202562 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.202571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.202966 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.203340 (+ 374us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.203949 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":340,"threads_started":1}
I20260924 10:02:15.207515 20633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.61@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
I20260924 10:02:15.219170 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.208304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40742)
0924 10:02:15.208433 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.208449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.208550 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.208795 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.208799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.209174 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.209316 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.209325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.210245 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.210250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.211173 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.211185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.211792 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.211797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.211951 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.212533 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.212550 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.214559 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.216992 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.216997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.217002 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.217347 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.217662 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.217667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.217671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.217784 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.218250 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.218254 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.218411 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.218531 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.218801 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 10:02:15.219221 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.208261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.13.162.61:45497)
0924 10:02:15.208424 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.208430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.208498 (+ 68us) server_negotiation.cc:407] Connection header received
0924 10:02:15.208640 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.208645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.208694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.208762 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.209430 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.210116 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.211291 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.211670 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.211773 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.214717 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.214721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.214724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.214751 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.216878 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.217445 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.217451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.217454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.217549 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.217872 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.217878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.217881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.218146 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.218266 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.218626 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.218928 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":45}
I20260924 10:02:15.223829 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Bootstrap starting.
I20260924 10:02:15.226868 20633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:15.227857 20633 log.cc:824] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:15.230448 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: No bootstrap required, opened a new log
I20260924 10:02:15.237581 20633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } }
I20260924 10:02:15.237880 20633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:15.237967 20633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7af120325d4809b3aff9cd14e1026c, State: Initialized, Role: FOLLOWER
I20260924 10:02:15.238585 20633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } }
I20260924 10:02:15.238688 20428 heartbeater.cc:503] Master 127.13.162.62:44465 was elected leader, sending a full tablet report...
I20260924 10:02:15.239439 20639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } } }
I20260924 10:02:15.239743 20639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:15.240612 20633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:02:15.248638 20650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:02:15.248755 20650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:02:15.251493 20566 master_runner.cc:428] Detected that this master 4b7af120325d4809b3aff9cd14e1026c is joining an existing cluster
I20260924 10:02:15.251624 20566 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.162.61:45497
I20260924 10:02:15.271728 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.256871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37236)
0924 10:02:15.257511 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.257525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.257648 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.257940 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.257944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.258558 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.258927 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.258938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.260250 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.260255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.261913 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.261928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.263059 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.263065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.263277 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.263890 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.263916 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.265147 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.268768 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.268773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.268777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.269282 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.269719 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.269723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.269727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.269901 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.270619 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.270624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.270830 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.270953 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.271273 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":25,"thread_start_us":346,"threads_started":1}
I20260924 10:02:15.272064 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.255842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.13.162.61:45497)
0924 10:02:15.256016 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.256023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.256767 (+ 744us) server_negotiation.cc:407] Connection header received
0924 10:02:15.256963 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.256968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.257042 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.257161 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.258684 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.259853 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.261955 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.262677 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.262773 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.265374 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.265381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.265385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.265417 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.269067 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.269877 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.269882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.269885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.270035 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.270467 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.270471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.270475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.270857 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.270971 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.271456 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.271696 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:02:15.272701 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.256009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.13.162.62:44465)
0924 10:02:15.256189 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.256195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.257608 (+ 1413us) server_negotiation.cc:407] Connection header received
0924 10:02:15.257757 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.257769 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.257828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.257905 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.259089 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.260044 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.262083 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.262689 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.262785 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.265357 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.265363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.265367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.265406 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.268608 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.269415 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.269420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.269423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.269577 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.270035 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.270039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.270042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.270418 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.270559 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.272183 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.272389 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:02:15.272944 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.255831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40750)
0924 10:02:15.256649 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.256668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.256835 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.257323 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.257327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.258041 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.258514 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.258531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.260066 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.260071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.261814 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.261828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.263098 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.263104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.263327 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.264027 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.264050 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.265193 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.269217 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.269222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.269225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.269747 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.270173 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.270178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.270182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.270363 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.271053 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.271058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.271204 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.271772 (+ 568us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.272521 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":22,"thread_start_us":517,"threads_started":1}
WARNING: no policy specified for oryx/127.13.162.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.60@KRBTEST.COM" created.
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244133, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.62@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
I20260924 10:02:15.302418 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.280111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40764)
0924 10:02:15.287379 (+ 7268us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.287420 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.287702 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.288266 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.288273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.288968 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.289169 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.289179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.290159 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.290165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.291227 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.291242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.291989 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.291995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.292189 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.292808 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.292826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.295258 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.299344 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.299359 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.299368 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.299939 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.300377 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.300381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.300385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.300547 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.301169 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.301178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.301391 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.301602 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.301953 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7018,"thread_start_us":6865,"threads_started":1}
I20260924 10:02:15.303370 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.280110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.13.162.61:45497)
0924 10:02:15.280300 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.280306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.287544 (+ 7238us) server_negotiation.cc:407] Connection header received
0924 10:02:15.287958 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.287962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.288086 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.288181 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.289323 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.290023 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.291366 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.291828 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.291938 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.295490 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.295498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.295502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.295541 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.299170 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.300086 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.300091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.300095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.300258 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.300671 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.300676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.300680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.301027 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.301168 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.302787 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.303032 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:02:15.305485 20104 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.162.61:45497: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c"
member_type: NON_VOTER
last_known_addr {
host: "127.13.162.61"
port: 45497
}
attrs {
promote: true
}
}
dest_uuid: "643687b7468c438ebc39c28da7f05a82"
cas_config_opid_index: -1
I20260924 10:02:15.306692 20104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } }
I20260924 10:02:15.310032 20663 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4b7af120325d4809b3aff9cd14e1026c (127.13.162.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } }
I20260924 10:02:15.310782 20608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.60.keytab.
Entry for principal oryx/127.13.162.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.60.keytab.
I20260924 10:02:15.313628 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:33795
--webserver_interface=127.13.162.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.60.keytab
--principal=oryx/127.13.162.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:02:15.314190 20608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Refusing update from remote peer 643687b7468c438ebc39c28da7f05a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 10:02:15.314210 20663 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.162.61:45497
I20260924 10:02:15.314688 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } }
I20260924 10:02:15.314884 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:15.315002 20566 master.cc:537] Master@127.13.162.61:45497 shutting down...
I20260924 10:02:15.316845 20662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:02:15.320477 20091 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 10:02:15.327775 20566 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:15.328214 20566 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:15.328404 20566 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: stopping tablet replica
I20260924 10:02:15.334830 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.334366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40776)
0924 10:02:15.334523 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.334643 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:45497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 10:02:15.335114 20091 consensus_peers.cc:487] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 -> Peer 4b7af120325d4809b3aff9cd14e1026c (127.13.162.61:45497): Couldn't send request to peer 4b7af120325d4809b3aff9cd14e1026c. Status: Network error: Client connection negotiation failed: client connection to 127.13.162.61:45497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:02:15.335721 20566 master.cc:552] Master@127.13.162.61:45497 shutdown complete.
I20260924 10:02:15.335809 20566 master_runner.cc:305] Clearing existing system tablet
I20260924 10:02:15.336216 20566 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:15.336316 20566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:15.339365 20566 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:02:15.341012 20668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:15.341406 20566 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:15.341722 20566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:02:15.341807 20566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
uuid: "4b7af120325d4809b3aff9cd14e1026c"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "b15657ed675af3cda80a3e3ba29fe7b9"
server_key_iv: "c0d5a608325da2cafeff8de022079ce3"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.347009 20566 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:15.354084 20566 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:02:15.354270 20566 log.cc:1196] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 10:02:15.354769 20566 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Deleting consensus metadata
I20260924 10:02:15.355006 20566 master_runner.cc:315] Copying system tablet from 127.13.162.62:44465
I20260924 10:02:15.359730 20566 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: tablet copy: Beginning tablet copy session from remote peer at address 127.13.162.62:44465
I20260924 10:02:15.376885 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.360628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.13.162.62:44465)
0924 10:02:15.360808 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.360813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.362829 (+ 2016us) server_negotiation.cc:407] Connection header received
0924 10:02:15.363116 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.363122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.363222 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.363334 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.364349 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.365460 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.367226 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.367899 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.368030 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.370451 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.370458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.370463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.370504 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.373909 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.374771 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.374777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.374781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.374923 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.375347 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.375353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.375356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.375720 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.375869 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.376303 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.376535 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:02:15.376942 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.360443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37238)
0924 10:02:15.362712 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.362732 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.362947 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.363382 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.363387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.363905 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.364167 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.364180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.365632 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.365637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.367064 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.367082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.368074 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.368081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.368359 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.369084 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.369105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.370194 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.374059 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.374065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.374069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.374629 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.375038 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.375042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.375046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.375234 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.375845 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.375850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.376032 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.376186 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.376510 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"thread_start_us":1999,"threads_started":1}
I20260924 10:02:15.378145 20125 tablet_copy_service.cc:140] P 643687b7468c438ebc39c28da7f05a82: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4b7af120325d4809b3aff9cd14e1026c ({username='oryx', principal='oryx/127.13.162.61@KRBTEST.COM'} at 127.0.0.1:37238)
I20260924 10:02:15.378469 20125 tablet_copy_service.cc:161] P 643687b7468c438ebc39c28da7f05a82: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4b7af120325d4809b3aff9cd14e1026c at {username='oryx', principal='oryx/127.13.162.61@KRBTEST.COM'} at 127.0.0.1:37238: session id = 4b7af120325d4809b3aff9cd14e1026c-00000000000000000000000000000000
I20260924 10:02:15.382923 20125 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:02:15.385489 20566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:02:15.392393 20566 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: tablet copy: Starting download of 0 data blocks...
I20260924 10:02:15.392732 20566 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: tablet copy: Starting download of 1 WAL segments...
I20260924 10:02:15.395576 20566 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:02:15.400074 20125 tablet_copy_service.cc:342] P 643687b7468c438ebc39c28da7f05a82: Request end of tablet copy session 4b7af120325d4809b3aff9cd14e1026c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.162.61@KRBTEST.COM'} at 127.0.0.1:37238
I20260924 10:02:15.400311 20125 tablet_copy_service.cc:433] P 643687b7468c438ebc39c28da7f05a82: ending tablet copy session 4b7af120325d4809b3aff9cd14e1026c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4b7af120325d4809b3aff9cd14e1026c
I20260924 10:02:15.412119 20566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:15.414866 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:15.415735 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:15.426625 20566 server_base.cc:1034] running on GCE node
W20260924 10:02:15.427830 20678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:15.428362 20566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:15.429113 20566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:15.430358 20566 hybrid_clock.cc:648] HybridClock initialized: now 1790244135430280 us; error 37 us; skew 500 ppm
I20260924 10:02:15.430600 20566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:15.432858 20566 init.cc:377] Logged in from keytab as oryx/127.13.162.61@KRBTEST.COM (short username oryx)
I20260924 10:02:15.438097 20566 webserver.cc:460] Webserver started at http://127.13.162.61:38103/ using document root <none> and password file <none>
I20260924 10:02:15.438482 20566 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:15.438558 20566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:15.441684 20566 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:02:15.443603 20685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:15.444016 20566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:02:15.444149 20566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
uuid: "4b7af120325d4809b3aff9cd14e1026c"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "b15657ed675af3cda80a3e3ba29fe7b9"
server_key_iv: "c0d5a608325da2cafeff8de022079ce3"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.444308 20566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:15.468461 20566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:15.472681 20566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:15.501691 20566 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.61:45497
I20260924 10:02:15.501732 20736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.61:45497 every 8 connection(s)
I20260924 10:02:15.502676 20566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
I20260924 10:02:15.508237 20737 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:02:15.512537 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Bootstrap starting.
I20260924 10:02:15.529363 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:02:15.530009 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Bootstrap complete.
I20260924 10:02:15.531226 20737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } }
I20260924 10:02:15.531499 20737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7af120325d4809b3aff9cd14e1026c, State: Initialized, Role: LEARNER
I20260924 10:02:15.531813 20737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } }
I20260924 10:02:15.532665 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } }
I20260924 10:02:15.532960 20737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:15.533181 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: LEARNER
I20260924 10:02:15.542418 20751 catalog_manager.cc:1259] Loaded cluster ID: e2e4455f26a84d5aabfb9cb777021f6b
I20260924 10:02:15.542541 20751 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: loading cluster ID for follower catalog manager: success
I20260924 10:02:15.546067 20751 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: acquiring CA information for follower catalog manager: success
I20260924 10:02:15.548497 20751 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:02:15.691296 20665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:15.691591 20665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:15.691722 20665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:15.699121 20665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:15.699249 20665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:15.699325 20665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:15.699397 20665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:15.699462 20665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:15.709610 20665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.60.keytab
--principal=oryx/127.13.162.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.162.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:15.711024 20665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:15.712388 20665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:02:15.712448 20561 heartbeater.cc:503] Master 127.13.162.62:44465 was elected leader, sending a full tablet report...
W20260924 10:02:15.723994 20757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:15.724340 20759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:15.724097 20756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:15.725157 20665 server_base.cc:1034] running on GCE node
I20260924 10:02:15.725900 20665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:15.729148 20665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:15.730397 20665 hybrid_clock.cc:648] HybridClock initialized: now 1790244135730348 us; error 40 us; skew 500 ppm
I20260924 10:02:15.730774 20665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:15.733961 20665 init.cc:377] Logged in from keytab as oryx/127.13.162.60@KRBTEST.COM (short username oryx)
I20260924 10:02:15.735674 20665 webserver.cc:460] Webserver started at http://127.13.162.60:46623/ using document root <none> and password file <none>
I20260924 10:02:15.736370 20665 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:15.736491 20665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:15.736780 20665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:15.739948 20665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/instance:
uuid: "e6cad33b49a1472489cdfbc23b253451"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "c979917d1cb0aff4cb5372e84b6b9fef"
server_key_iv: "49d242c0e416d5fa473b3bd3236f7362"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.740706 20665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal/instance:
uuid: "e6cad33b49a1472489cdfbc23b253451"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "c979917d1cb0aff4cb5372e84b6b9fef"
server_key_iv: "49d242c0e416d5fa473b3bd3236f7362"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.746089 20665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:02:15.749660 20766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:15.751259 20665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:15.751430 20665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
uuid: "e6cad33b49a1472489cdfbc23b253451"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "c979917d1cb0aff4cb5372e84b6b9fef"
server_key_iv: "49d242c0e416d5fa473b3bd3236f7362"
server_key_version: "encryptionkey@0"
I20260924 10:02:15.751634 20665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:15.788579 20665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:15.793133 20665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:15.793433 20665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:15.810516 20665 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.60:33795
I20260924 10:02:15.810537 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.60:33795 every 8 connection(s)
I20260924 10:02:15.811878 20665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
I20260924 10:02:15.816224 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:15.821599 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.815792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40780)
0924 10:02:15.815973 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.815990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.816091 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.816787 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.816791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.816818 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:15.816976 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.816998 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.818361 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.818375 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.819591 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.819605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.820752 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.820770 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.820982 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.821050 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.821157 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.821307 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:02:15.821609 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.815953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.13.162.61:45497)
0924 10:02:15.816459 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.816464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.816479 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:02:15.816536 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.816540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.816605 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.816725 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:15.817147 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.818134 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.819758 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.820536 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.820666 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.820875 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.821245 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.821355 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260924 10:02:15.821990 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20665
I20260924 10:02:15.823171 20712 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 10:02:15.823908 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643687b7468c438ebc39c28da7f05a82. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } }
I20260924 10:02:15.824162 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: LEARNER
I20260924 10:02:15.823722 20818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.60@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.60@KRBTEST.COM
I20260924 10:02:15.843386 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.826747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.13.162.62:44465)
0924 10:02:15.826921 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.826926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.827691 (+ 765us) server_negotiation.cc:407] Connection header received
0924 10:02:15.828197 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.828202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.828273 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.828382 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.830045 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.831172 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.832803 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.833238 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.833348 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.836803 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.836808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.836811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.836844 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.839945 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.840852 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.840856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.840858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.840973 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.841411 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.841416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.841419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.841772 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.841894 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.842769 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.842993 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:02:15.844724 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.825141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.13.162.60:33795)
0924 10:02:15.825956 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.825968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.826006 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:02:15.827978 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.828000 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.828299 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.828490 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.830189 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.831776 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.833084 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.833663 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.833787 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.837851 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.837864 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.837882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.837924 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.841108 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.841835 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.841844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.841847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.841992 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.842355 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.842359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.842362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.842758 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.843035 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.843400 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.843612 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260924 10:02:15.844741 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.826730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37250)
0924 10:02:15.827524 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.827580 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.827976 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.828430 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.828438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.829346 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.829868 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.829892 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.831361 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.831365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.832659 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.832674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.833458 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.833466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.833656 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.834442 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.834463 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.836597 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.840075 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.840084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.840099 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.840717 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.841077 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.841083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.841087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.841306 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.841879 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.841888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.842348 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.842697 (+ 349us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.843207 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":394,"threads_started":1}
I20260924 10:02:15.847121 20818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.60@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
I20260924 10:02:15.859495 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.847876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40788)
0924 10:02:15.848177 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.848193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.848283 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.848523 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.848526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.848862 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.849001 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.849008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.850024 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.850027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.851132 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.851144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.851902 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.851906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.852032 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.852556 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.852572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.854501 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.857436 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.857440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.857443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.857832 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.858193 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.858197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.858199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.858370 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.858829 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.858832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.858955 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.859053 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.859268 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 10:02:15.859562 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.847891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.13.162.61:45497)
0924 10:02:15.848124 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.848129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.848234 (+ 105us) server_negotiation.cc:407] Connection header received
0924 10:02:15.848361 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.848364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.848423 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.848500 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.849173 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.849902 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.851309 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.851749 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.851862 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.854686 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.854691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.854694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.854729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.857314 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.857933 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.857937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.857939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.858063 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.858458 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.858462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.858464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.858742 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.858854 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.859147 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.859334 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 10:02:15.860592 20818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244135, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.60@KRBTEST.COM for oryx/127.13.162.60@KRBTEST.COM
I20260924 10:02:15.872360 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.861315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.13.162.60:33795)
0924 10:02:15.861440 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.861445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.861460 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:02:15.861620 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.861625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.861677 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.861751 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.862448 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.863139 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.864178 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.864557 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.864668 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.867692 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.867696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.867699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.867726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.869964 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.870566 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.870571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.870574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.870681 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.871045 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.871048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.871050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.871297 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.871415 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.871757 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.872030 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":45}
I20260924 10:02:15.872380 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.861223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.0.0.1:50766)
0924 10:02:15.861401 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.861420 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.861514 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.861778 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.861782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.862161 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.862301 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.862308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.863248 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.863253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.864052 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.864062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.864678 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.864683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.864913 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.865435 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.865452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.867515 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.870090 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.870095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.870100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.870470 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.870793 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.870798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.870802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.870944 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.871396 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.871400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.871544 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.871647 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.871911 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":20}
I20260924 10:02:15.877236 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Bootstrap starting.
I20260924 10:02:15.880370 20818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:15.881345 20818 log.cc:824] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:15.884048 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: No bootstrap required, opened a new log
I20260924 10:02:15.891050 20818 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } }
I20260924 10:02:15.891372 20818 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:15.891455 20818 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cad33b49a1472489cdfbc23b253451, State: Initialized, Role: FOLLOWER
I20260924 10:02:15.891985 20818 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } }
I20260924 10:02:15.892751 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } } }
I20260924 10:02:15.893056 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:15.893748 20818 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:15.899652 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20166
W20260924 10:02:15.903963 20836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:02:15.904137 20836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:02:15.906577 20665 master_runner.cc:428] Detected that this master e6cad33b49a1472489cdfbc23b253451 is joining an existing cluster
I20260924 10:02:15.906708 20665 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.162.60:33795
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.1@KRBTEST.COM
I20260924 10:02:15.923921 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.901364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.13.162.1:45073)
0924 10:02:15.902939 (+ 1575us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.902945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.902968 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:02:15.903019 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.903025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.903273 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.903485 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.905049 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.906690 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.908532 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.909358 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.909454 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.915985 (+ 6531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.916010 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.916030 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.916080 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.920037 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.921046 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.921058 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.921062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.921238 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.921790 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.921795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.921798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.922268 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.922576 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.923222 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.923510 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":1284,"threads_started":1}
I20260924 10:02:15.926566 20289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:15.939793 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.911413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.13.162.62:44465)
0924 10:02:15.911590 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.911596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.923242 (+ 11646us) server_negotiation.cc:407] Connection header received
0924 10:02:15.923860 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.923867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.923965 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.924034 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.925481 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.926689 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.928815 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.929499 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.929601 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.932552 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.932559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.932564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.932627 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.936186 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.937143 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.937150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.937154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.937352 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.937984 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.937989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.937992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.938432 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.938551 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.939224 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.939446 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:02:15.941139 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.910989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37264)
0924 10:02:15.923101 (+ 12112us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.923127 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.923693 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.924252 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.924257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.924931 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.925304 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.925316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.926914 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.926921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.928628 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.928645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.929960 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.929968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.930156 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.930984 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.931010 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.932287 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.936431 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.936437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.936442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.936967 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.937607 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.937613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.937617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.937830 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.938655 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.938659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.938880 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.940333 (+ 1453us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.940682 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11919,"thread_start_us":28332,"threads_started":1}
Sep 24 10:02:15 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244133, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.62@KRBTEST.COM for oryx/127.13.162.60@KRBTEST.COM
I20260924 10:02:15.966642 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.912123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.13.162.60:33795)
0924 10:02:15.912314 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.912321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.938358 (+ 26037us) server_negotiation.cc:407] Connection header received
0924 10:02:15.941476 (+ 3118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.941480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.941550 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.941613 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.942992 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.944057 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.953393 (+ 9336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.954129 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.954249 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.959118 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.959124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.959129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.959158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.962753 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.964490 (+ 1737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.964495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.964499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.964641 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.965021 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.965026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.965029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.965383 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.965538 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.966023 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.966253 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 10:02:15.967871 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.911544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.0.0.1:50774)
0924 10:02:15.938246 (+ 26702us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.938266 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.939789 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.941899 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.941904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.942515 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.942855 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.942868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.945528 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.945532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.952595 (+ 7063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.952607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.954584 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.954590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.954790 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.955485 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.955508 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.958971 (+ 3463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.962902 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.962907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.962912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.964371 (+ 1459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.964781 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.964785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.964789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.964923 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.965606 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.965610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.965771 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.967136 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.967430 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26543,"thread_start_us":11725,"threads_started":1}
I20260924 10:02:15.970661 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.911296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.13.162.61:45497)
0924 10:02:15.912012 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.912018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.938906 (+ 26888us) server_negotiation.cc:407] Connection header received
0924 10:02:15.939929 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.939935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.940014 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.940075 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.944270 (+ 4195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.945281 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.952396 (+ 7115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.953075 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.953198 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.963421 (+ 10223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.963428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.963433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.963475 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.966717 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.967494 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.967500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.967503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.967648 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.968069 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.968074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.968078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.968395 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.968511 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.970092 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.970336 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579}
I20260924 10:02:15.971823 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.911656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40798)
0924 10:02:15.938808 (+ 27152us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.938826 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.939633 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.940268 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.940272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.940994 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.941301 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.941313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.947174 (+ 5861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.947179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.952240 (+ 5061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.952259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.956786 (+ 4527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.956793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.957013 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.957605 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.957623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.958764 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.966872 (+ 8108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.966878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.966882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.967357 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.967798 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.967803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.967807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.967953 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.968617 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.968622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.969830 (+ 1208us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.971071 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.971389 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26991,"mutex_wait_us":142,"thread_start_us":38153,"threads_started":1}
I20260924 10:02:15.974129 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.947311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.0.0.1:50788)
0924 10:02:15.947522 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:15.947541 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:15.947673 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:15.953568 (+ 5895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:15.953575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:15.954073 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:15.954303 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.954342 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.956289 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.956295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.957723 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:15.957739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.958856 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.958863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:15.959021 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.959558 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:15.959580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:15.962237 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:15.971056 (+ 8819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.971062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.971066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.971578 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.972025 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:15.972030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:15.972034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:15.972209 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:15.973230 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:15.973235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:15.973396 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 10:02:15.973536 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 10:02:15.973798 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:02:15.975184 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:15.947285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.13.162.60:33795)
0924 10:02:15.953128 (+ 5843us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:15.953133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:15.953155 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:15.953267 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:15.953271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:15.953369 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:15.954491 (+ 1122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:15.954824 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.956112 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.957907 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:15.958672 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:15.959311 (+ 639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:15.962510 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:15.962518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:15.962523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:15.962564 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:15.970898 (+ 8334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.971735 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.971740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.971743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.971895 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:15.972352 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:15.972356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:15.972360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:15.973078 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:15.974505 (+ 1427us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:15.974621 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 10:02:15.974807 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5691,"thread_start_us":7119,"threads_started":1}
I20260924 10:02:15.976694 20104 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.162.60:33795: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e6cad33b49a1472489cdfbc23b253451"
member_type: NON_VOTER
last_known_addr {
host: "127.13.162.60"
port: 33795
}
attrs {
promote: true
}
}
dest_uuid: "643687b7468c438ebc39c28da7f05a82"
cas_config_opid_index: 5
I20260924 10:02:15.977733 20104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } }
I20260924 10:02:15.979403 20662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e6cad33b49a1472489cdfbc23b253451 (127.13.162.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:15.979954 20712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 LEARNER]: Refusing update from remote peer 643687b7468c438ebc39c28da7f05a82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:02:15.981244 20793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:15.983062 20662 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.162.60:33795
I20260924 10:02:15.995003 20849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 10:02:16.004020 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.005286 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:16.004923 20712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e6cad33b49a1472489cdfbc23b253451 (127.13.162.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.009752 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: NON_VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: true } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.010021 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: LEARNER
I20260924 10:02:16.010671 20665 master.cc:537] Master@127.13.162.60:33795 shutting down...
I20260924 10:02:16.010732 20793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Refusing update from remote peer 643687b7468c438ebc39c28da7f05a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:02:16.011747 20662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:16.020426 20662 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: attempting to promote NON_VOTER 4b7af120325d4809b3aff9cd14e1026c to VOTER
I20260924 10:02:16.021194 20662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } }
I20260924 10:02:16.023748 20665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:16.023991 20711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 LEARNER]: Refusing update from remote peer 643687b7468c438ebc39c28da7f05a82: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 10:02:16.024325 20665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:16.024576 20665 tablet_replica.cc:330] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: stopping tablet replica
I20260924 10:02:16.024621 20662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 10:02:16.027652 20711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7af120325d4809b3aff9cd14e1026c (127.13.162.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.030460 20662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7af120325d4809b3aff9cd14e1026c (127.13.162.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.032528 20665 master.cc:552] Master@127.13.162.60:33795 shutdown complete.
I20260924 10:02:16.032694 20665 master_runner.cc:305] Clearing existing system tablet
I20260924 10:02:16.033121 20665 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:16.033262 20665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:16.035396 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.035645 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:16.036690 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.036583 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.037045 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:16.037156 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:16.043752 20665 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.008s
W20260924 10:02:16.044052 20090 connection.cc:698] client connection to 127.13.162.60:33795 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260924 10:02:16.045682 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.045156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.0.0.1:50802)
0924 10:02:16.045390 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:16.045482 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260924 10:02:16.046089 20090 consensus_peers.cc:487] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 -> Peer e6cad33b49a1472489cdfbc23b253451 (127.13.162.60:33795): Couldn't send request to peer e6cad33b49a1472489cdfbc23b253451. Status: Network error: Client connection negotiation failed: client connection to 127.13.162.60:33795: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:02:16.050717 20855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:16.051054 20665 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:16.051457 20665 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.006s
I20260924 10:02:16.051545 20665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
uuid: "e6cad33b49a1472489cdfbc23b253451"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "c979917d1cb0aff4cb5372e84b6b9fef"
server_key_iv: "49d242c0e416d5fa473b3bd3236f7362"
server_key_version: "encryptionkey@0"
I20260924 10:02:16.054929 20665 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:16.059485 20665 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:02:16.059639 20665 log.cc:1196] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 10:02:16.060042 20665 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Deleting consensus metadata
I20260924 10:02:16.060258 20665 master_runner.cc:315] Copying system tablet from 127.13.162.62:44465
I20260924 10:02:16.062539 20665 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: tablet copy: Beginning tablet copy session from remote peer at address 127.13.162.62:44465
I20260924 10:02:16.081493 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.063270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.13.162.62:44465)
0924 10:02:16.063486 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:16.063493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:16.070863 (+ 7370us) server_negotiation.cc:407] Connection header received
0924 10:02:16.071039 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:16.071044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:16.071124 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:16.071210 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:16.072062 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.072781 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.073999 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.074434 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.074545 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.076318 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:16.076323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:16.076326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:16.076359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:16.079043 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.079658 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.079663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.079667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.079795 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.080142 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.080146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.080148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.080420 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:16.080557 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:16.080923 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:02:16.081133 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:02:16.081547 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.063201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37268)
0924 10:02:16.070732 (+ 7531us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:16.070753 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:16.070914 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:16.071320 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:16.071325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:16.071729 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:16.071917 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.071926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.072921 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.072926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.073861 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.073873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.074586 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.074592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.074775 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.075365 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:16.075381 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:16.076111 (+ 730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:16.079165 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.079171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.079175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.079548 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.079897 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.079902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.079906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.080045 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.080527 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:16.080532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:16.080687 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 10:02:16.080819 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 10:02:16.081134 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7390,"thread_start_us":7259,"threads_started":1}
I20260924 10:02:16.082438 20125 tablet_copy_service.cc:140] P 643687b7468c438ebc39c28da7f05a82: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e6cad33b49a1472489cdfbc23b253451 ({username='oryx', principal='oryx/127.13.162.60@KRBTEST.COM'} at 127.0.0.1:37268)
I20260924 10:02:16.082602 20125 tablet_copy_service.cc:161] P 643687b7468c438ebc39c28da7f05a82: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e6cad33b49a1472489cdfbc23b253451 at {username='oryx', principal='oryx/127.13.162.60@KRBTEST.COM'} at 127.0.0.1:37268: session id = e6cad33b49a1472489cdfbc23b253451-00000000000000000000000000000000
I20260924 10:02:16.084892 20125 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:02:16.086457 20665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:02:16.091766 20665 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: tablet copy: Starting download of 0 data blocks...
I20260924 10:02:16.092031 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20166
I20260924 10:02:16.092136 20665 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: tablet copy: Starting download of 1 WAL segments...
I20260924 10:02:16.094204 20665 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:02:16.097499 20125 tablet_copy_service.cc:342] P 643687b7468c438ebc39c28da7f05a82: Request end of tablet copy session e6cad33b49a1472489cdfbc23b253451-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.162.60@KRBTEST.COM'} at 127.0.0.1:37268
I20260924 10:02:16.097612 20125 tablet_copy_service.cc:433] P 643687b7468c438ebc39c28da7f05a82: ending tablet copy session e6cad33b49a1472489cdfbc23b253451-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e6cad33b49a1472489cdfbc23b253451
I20260924 10:02:16.107190 20665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:16.110056 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:16.110751 20863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:16.114120 20865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:16.114301 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:45073
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:44465
--webserver_port=42423
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:02:16.115057 20665 server_base.cc:1034] running on GCE node
I20260924 10:02:16.115419 20665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:16.115999 20665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:16.119287 20665 hybrid_clock.cc:648] HybridClock initialized: now 1790244136119232 us; error 31 us; skew 500 ppm
I20260924 10:02:16.119520 20665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:16 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244136, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:16.121388 20665 init.cc:377] Logged in from keytab as oryx/127.13.162.60@KRBTEST.COM (short username oryx)
I20260924 10:02:16.122632 20665 webserver.cc:460] Webserver started at http://127.13.162.60:37321/ using document root <none> and password file <none>
I20260924 10:02:16.122972 20665 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:16.123070 20665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:16.126029 20665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 10:02:16.127914 20873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:16.128371 20665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 10:02:16.128527 20665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
uuid: "e6cad33b49a1472489cdfbc23b253451"
format_stamp: "Formatted at 2026-09-24 10:02:15 on dist-test-slave-jvsj"
server_key: "c979917d1cb0aff4cb5372e84b6b9fef"
server_key_iv: "49d242c0e416d5fa473b3bd3236f7362"
server_key_version: "encryptionkey@0"
I20260924 10:02:16.128782 20665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:16.163591 20665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:16.167752 20665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:16.190651 20665 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.60:33795
I20260924 10:02:16.190694 20924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.60:33795 every 8 connection(s)
I20260924 10:02:16.191645 20665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
I20260924 10:02:16.196635 20925 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:02:16.200443 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Bootstrap starting.
I20260924 10:02:16.219404 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:02:16.219955 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: Bootstrap complete.
I20260924 10:02:16.221063 20925 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } }
I20260924 10:02:16.221379 20925 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6cad33b49a1472489cdfbc23b253451, State: Initialized, Role: LEARNER
I20260924 10:02:16.221671 20925 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } }
I20260924 10:02:16.222854 20925 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:16.222684 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.223268 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:02:16.232050 20939 catalog_manager.cc:1259] Loaded cluster ID: e2e4455f26a84d5aabfb9cb777021f6b
I20260924 10:02:16.232160 20939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: loading cluster ID for follower catalog manager: success
I20260924 10:02:16.235950 20939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: acquiring CA information for follower catalog manager: success
I20260924 10:02:16.238729 20939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:02:16.475014 20868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:16.475317 20868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:16.475456 20868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:16.483067 20868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:16.483192 20868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:16.483358 20868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:16.493870 20868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:45073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=42423
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:16.495373 20868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:16.496650 20868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:16.510897 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:16.511008 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:16.511711 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:16.512187 20868 server_base.cc:1034] running on GCE node
I20260924 10:02:16.512856 20868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:16.513870 20868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:16.515120 20868 hybrid_clock.cc:648] HybridClock initialized: now 1790244136515041 us; error 67 us; skew 500 ppm
I20260924 10:02:16.515475 20868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:16 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244136, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:16.518821 20868 init.cc:377] Logged in from keytab as oryx/127.13.162.1@KRBTEST.COM (short username oryx)
I20260924 10:02:16.520327 20868 webserver.cc:460] Webserver started at http://127.13.162.1:42423/ using document root <none> and password file <none>
I20260924 10:02:16.521016 20868 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:16.521145 20868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:16.527035 20868 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:02:16.530223 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:16.531912 20868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:16.532119 20868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "9fa49ed9187a42568c7ff491e12fb40b"
format_stamp: "Formatted at 2026-09-24 10:02:13 on dist-test-slave-jvsj"
server_key: "eeb7501a4c3073da9aae28760bb61404"
server_key_iv: "ad2d6cdfff08a147fed2ca0e72804f27"
server_key_version: "encryptionkey@0"
I20260924 10:02:16.533208 20868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:16.563282 20868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:16.567564 20868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:16.567803 20868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:16.568717 20868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:16.570116 20868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:16.570201 20868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:16.570272 20868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:16.570349 20868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:16.600284 20868 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:45073
I20260924 10:02:16.600330 21066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:45073 every 8 connection(s)
I20260924 10:02:16.601680 20868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:16.610195 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 20868
I20260924 10:02:16.610570 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20300
Sep 24 10:02:16 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.2@KRBTEST.COM
Sep 24 10:02:16 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244136, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
Sep 24 10:02:16 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244136, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.60@KRBTEST.COM
Sep 24 10:02:16 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244136, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
I20260924 10:02:16.659428 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.611954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.13.162.2:44257)
0924 10:02:16.624495 (+ 12541us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:16.624508 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:16.629518 (+ 5010us) server_negotiation.cc:407] Connection header received
0924 10:02:16.629618 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:16.629629 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:16.629975 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:16.630232 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:16.634027 (+ 3795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.637208 (+ 3181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.639399 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.640327 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.640460 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.646747 (+ 6287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:16.646764 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:16.646783 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:16.646832 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:16.651548 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.652782 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.652791 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.652795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.652937 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.656154 (+ 3217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.656160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.656164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.657119 (+ 955us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:16.658571 (+ 1452us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:16.658761 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:02:16.659033 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12334,"thread_start_us":12578,"threads_started":1}
I20260924 10:02:16.662338 20423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:16.662566 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.604773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:48733 (local address 127.13.162.61:45497)
0924 10:02:16.616153 (+ 11380us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:16.616160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:16.625616 (+ 9456us) server_negotiation.cc:407] Connection header received
0924 10:02:16.633519 (+ 7903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:16.633527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:16.633658 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:16.633750 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:16.639325 (+ 5575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.640457 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.645722 (+ 5265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.646451 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.646567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.656044 (+ 9477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:16.656051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:16.656055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:16.656098 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:16.659330 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.660417 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.660423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.660426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.660569 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.660973 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.660977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.660981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.661294 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:16.661421 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:16.661895 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 10:02:16.662108 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11227,"thread_start_us":11113,"threads_started":1}
I20260924 10:02:16.663730 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.642862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.0.0.1:50808)
0924 10:02:16.654947 (+ 12085us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:16.654966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:16.655089 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:16.655757 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:16.655761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:16.655786 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:16.656066 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.656079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.657668 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.657674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.659510 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.659525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.662885 (+ 3360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.662893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.663116 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.663157 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:02:16.663255 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:02:16.663411 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11945,"thread_start_us":11791,"threads_started":1}
I20260924 10:02:16.663785 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.605299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.13.162.1:51747)
0924 10:02:16.637174 (+ 31875us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:16.637196 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:16.637340 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:16.638883 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:16.638889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:16.639499 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:16.639856 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.639867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.641458 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.641463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.645639 (+ 4176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.645652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.651242 (+ 5590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.651249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.651460 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.652011 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:16.652034 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:16.655307 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:16.658936 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.658947 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.658967 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.659664 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.660103 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.660108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.660121 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.660347 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.661017 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:16.661027 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:16.661482 (+ 455us) client_negotiation.cc:769] Sending connection context
0924 10:02:16.661784 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 10:02:16.662309 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31690,"mutex_wait_us":678,"spinlock_wait_cycles":1408,"thread_start_us":26891,"threads_started":1}
I20260924 10:02:16.664345 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.605299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.13.162.1:48733)
0924 10:02:16.625392 (+ 20093us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:16.625458 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:16.628302 (+ 2844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:16.634622 (+ 6320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:16.634638 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:16.635537 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:16.636472 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.636499 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.640727 (+ 4228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.640732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.644696 (+ 3964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.644713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.647032 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.647039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.647357 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.648818 (+ 1461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:16.648929 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:16.655785 (+ 6856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:16.659481 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.659742 (+ 261us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.659746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.660281 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.660718 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.660722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.660725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.660864 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.661491 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:16.661495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:16.661607 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 10:02:16.663133 (+ 1526us) client_negotiation.cc:240] Negotiation successful
0924 10:02:16.663438 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19739,"spinlock_wait_cycles":50432,"thread_start_us":19472,"threads_started":1}
I20260924 10:02:16.664558 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.643041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.13.162.60:33795)
0924 10:02:16.653441 (+ 10400us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:16.653448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:16.655211 (+ 1763us) server_negotiation.cc:407] Connection header received
0924 10:02:16.655280 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:16.655285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:16.655384 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:16.655502 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:16.656237 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.657513 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.659691 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.660953 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.661068 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.661312 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:16.663967 (+ 2655us) server_negotiation.cc:299] Negotiation successful
0924 10:02:16.664211 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10262,"thread_start_us":11835,"threads_started":1}
I20260924 10:02:16.665146 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.604540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51747 (local address 127.13.162.60:33795)
0924 10:02:16.638598 (+ 34058us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:16.638602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:16.638625 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:02:16.638672 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:16.638675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:16.638730 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:16.638828 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:16.640031 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.641302 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.649401 (+ 8099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.650090 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.651826 (+ 1736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.655530 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:16.655537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:16.655545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:16.655586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:16.658774 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.659835 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.659839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.659843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.659971 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.660467 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.660472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.660475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.660825 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:16.660944 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:16.664592 (+ 3648us) server_negotiation.cc:299] Negotiation successful
0924 10:02:16.664811 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33944,"thread_start_us":33827,"threads_started":1}
I20260924 10:02:16.666071 21069 heartbeater.cc:347] Connected to a master server at 127.13.162.61:45497
I20260924 10:02:16.666043 21067 heartbeater.cc:347] Connected to a master server at 127.13.162.60:33795
I20260924 10:02:16.666452 21069 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:16.666457 21067 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:16.667449 21069 heartbeater.cc:511] Master 127.13.162.61:45497 requested a full tablet report, sending...
I20260924 10:02:16.667476 21067 heartbeater.cc:511] Master 127.13.162.60:33795 requested a full tablet report, sending...
I20260924 10:02:16.668116 20900 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 10:02:16.668921 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643687b7468c438ebc39c28da7f05a82. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:16.669199 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:02:16.669978 20890 ts_manager.cc:194] Registered new tserver with Master: 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073)
I20260924 10:02:16.670814 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.605926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:41511)
0924 10:02:16.633332 (+ 27406us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:16.633356 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:16.633497 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:16.643093 (+ 9596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:16.643098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:16.643621 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:16.643900 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.643911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.645458 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.645467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.646921 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:16.646935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.648055 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.648062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:16.648243 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.648804 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:16.648834 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:16.651980 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:16.659691 (+ 7711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.659696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.659699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.660273 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.668038 (+ 7765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:16.668043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:16.668047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:16.668253 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:16.669721 (+ 1468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:16.669726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:16.669894 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 10:02:16.670042 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 10:02:16.670393 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27151,"thread_start_us":30873,"threads_started":1}
I20260924 10:02:16.670905 20702 ts_manager.cc:194] Registered new tserver with Master: 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073)
I20260924 10:02:16.671970 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:16.604537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:41511 (local address 127.13.162.62:44465)
0924 10:02:16.642738 (+ 38201us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:16.642744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:16.642765 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:02:16.642821 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:16.642827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:16.642908 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:16.644051 (+ 1143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:16.644202 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.645293 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.647233 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:16.647854 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:16.648517 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:16.652473 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:16.652479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:16.652484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:16.652517 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:16.655698 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.660410 (+ 4712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.660415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.660418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.660562 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:16.668485 (+ 7923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:16.668490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:16.668493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:16.668883 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:16.671190 (+ 2307us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:16.671331 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 10:02:16.671583 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38066,"thread_start_us":37950,"threads_started":1}
I20260924 10:02:16.672771 21068 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44465
I20260924 10:02:16.672902 21068 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:16.673336 21068 heartbeater.cc:511] Master 127.13.162.62:44465 requested a full tablet report, sending...
I20260924 10:02:16.674501 20104 ts_manager.cc:194] Re-registered known tserver with Master: 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073)
I20260924 10:02:16.675318 20104 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:41511
I20260924 10:02:16.759958 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20300
I20260924 10:02:16.773003 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:44257
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:44465
--webserver_port=42117
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:02:17.093410 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: NON_VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: true } } }
I20260924 10:02:17.093699 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:17.095191 21086 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: attempting to promote NON_VOTER e6cad33b49a1472489cdfbc23b253451 to VOTER
W20260924 10:02:17.095537 21082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:17.095819 21082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260924 10:02:17.095757 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } }
W20260924 10:02:17.095961 21082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:02:17.097643 20900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 LEARNER]: Refusing update from remote peer 643687b7468c438ebc39c28da7f05a82: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 10:02:17.097644 20712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Refusing update from remote peer 643687b7468c438ebc39c28da7f05a82: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 10:02:17.098194 21086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 10:02:17.098579 21087 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 10:02:17.101461 21086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e6cad33b49a1472489cdfbc23b253451 (127.13.162.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.102111 20712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e6cad33b49a1472489cdfbc23b253451 (127.13.162.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
W20260924 10:02:17.106380 21082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:17.106573 21082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:17.106824 21082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:17.106587 20900 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e6cad33b49a1472489cdfbc23b253451 (127.13.162.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.111238 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.111521 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:17.112742 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.113336 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:17.113929 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.114198 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:17.114506 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.114547 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "643687b7468c438ebc39c28da7f05a82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:17.114780 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:17.114882 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:17.120062 21082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:44257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=42117
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:17.121682 21082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:17.123044 21082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:17.136046 21096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:17.136336 21095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:17.136379 21098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:17.136984 21082 server_base.cc:1034] running on GCE node
I20260924 10:02:17.137581 21082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:17.138571 21082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:17.139771 21082 hybrid_clock.cc:648] HybridClock initialized: now 1790244137139720 us; error 48 us; skew 500 ppm
I20260924 10:02:17.140058 21082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:17.143100 21082 init.cc:377] Logged in from keytab as oryx/127.13.162.2@KRBTEST.COM (short username oryx)
I20260924 10:02:17.146865 21082 webserver.cc:460] Webserver started at http://127.13.162.2:42117/ using document root <none> and password file <none>
I20260924 10:02:17.147565 21082 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:17.147691 21082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:17.153213 21082 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:02:17.156006 21105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:17.157351 21082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:02:17.157496 21082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "82add254d9d8479480e7018a651d2291"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "339e14692eacb2d825436c3e9edc91b5"
server_key_iv: "3be672e9f6a8ed5f23848cc1f8ad15ea"
server_key_version: "encryptionkey@0"
I20260924 10:02:17.158468 21082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:17.192823 21082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:17.197064 21082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:17.197337 21082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:17.198266 21082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:17.199617 21082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:17.199702 21082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:17.199800 21082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:17.199863 21082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:17.227159 21082 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:44257
I20260924 10:02:17.227265 21217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:44257 every 8 connection(s)
I20260924 10:02:17.228482 21082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:17.232563 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 21082
I20260924 10:02:17.232874 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20433
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.60@KRBTEST.COM
I20260924 10:02:17.271005 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.232537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:32793 (local address 127.13.162.61:45497)
0924 10:02:17.240849 (+ 8312us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.240854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.240875 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:02:17.240942 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.240947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.241040 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.241164 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.253158 (+ 11994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.254572 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.257344 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.257988 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.258073 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.263880 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.263886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.263891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.263941 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.267371 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.268360 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.268364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.268367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.268509 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.269055 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.269060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.269064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.269404 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.269531 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.270440 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.270666 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8198,"thread_start_us":8109,"threads_started":1}
I20260924 10:02:17.272154 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.231795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.13.162.2:32793)
0924 10:02:17.232601 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.232777 (+ 176us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.235611 (+ 2834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.250977 (+ 15366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.250995 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.251919 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:17.252872 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.252904 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.254893 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.254899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.257176 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.257192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.258423 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.258430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.258802 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.259558 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:17.259580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:17.263625 (+ 4045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:17.267525 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.267534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.267551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.268215 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.268648 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.268652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.268655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.268922 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.269629 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:17.269637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:17.270058 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.270334 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.270848 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":5749,"threads_started":1}
I20260924 10:02:17.273480 21220 heartbeater.cc:347] Connected to a master server at 127.13.162.61:45497
I20260924 10:02:17.273762 21220 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:17.274501 21220 heartbeater.cc:511] Master 127.13.162.61:45497 requested a full tablet report, sending...
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.3@KRBTEST.COM
I20260924 10:02:17.276189 20702 ts_manager.cc:194] Registered new tserver with Master: 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257)
I20260924 10:02:17.279772 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.233451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:40203)
0924 10:02:17.261309 (+ 27858us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.261335 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.261442 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.261803 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.261808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.262303 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:17.262553 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.262561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.263859 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.263865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.265366 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.265379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.266539 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.266545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.266710 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.267241 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:17.267261 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:17.270535 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:17.277338 (+ 6803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.277342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.277345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.277730 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.278160 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.278163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.278166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.278298 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.278817 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:17.278826 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:17.278960 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.279066 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.279351 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27739,"mutex_wait_us":25,"thread_start_us":27213,"threads_started":1}
I20260924 10:02:17.279799 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.232299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:40203 (local address 127.13.162.62:44465)
0924 10:02:17.240523 (+ 8224us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.240529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.261417 (+ 20888us) server_negotiation.cc:407] Connection header received
0924 10:02:17.261560 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.261566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.261661 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.261770 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.262697 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.263710 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.265524 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.266160 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.266255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.270780 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.270786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.270791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.270831 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.277209 (+ 6378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.277864 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.277870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.277874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.278026 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.278444 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.278450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.278453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.278702 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.278831 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.279254 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.279463 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8098,"thread_start_us":7993,"threads_started":1}
I20260924 10:02:17.280730 21219 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44465
I20260924 10:02:17.280861 21219 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:17.281364 21219 heartbeater.cc:511] Master 127.13.162.62:44465 requested a full tablet report, sending...
I20260924 10:02:17.281397 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.13.162.2:46207)
0924 10:02:17.260384 (+ 27286us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.260405 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.260552 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.261120 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.261126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.262856 (+ 1730us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:17.263212 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.263224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.264730 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.264735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.266235 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.266247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.267371 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.267377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.267564 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.268097 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:17.268124 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:17.271729 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:17.278341 (+ 6612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.278347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.278350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.278879 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.279297 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.279301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.279305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.279443 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.280441 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:17.280446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:17.280596 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.280741 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.281011 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27135,"thread_start_us":31782,"threads_started":1}
I20260924 10:02:17.282245 20104 ts_manager.cc:194] Re-registered known tserver with Master: 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257)
I20260924 10:02:17.282285 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.231203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:46207 (local address 127.13.162.60:33795)
0924 10:02:17.247728 (+ 16525us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.247735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.260527 (+ 12792us) server_negotiation.cc:407] Connection header received
0924 10:02:17.260694 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.260701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.260783 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.260893 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.263405 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.264561 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.266417 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.266995 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.267075 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.272736 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.272742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.272747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.272788 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.278204 (+ 5416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.279022 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.279027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.279031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.279175 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.279570 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.279575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.279578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.280246 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.280382 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.281741 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.281951 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16372,"thread_start_us":16513,"threads_started":1}
I20260924 10:02:17.283129 21218 heartbeater.cc:347] Connected to a master server at 127.13.162.60:33795
I20260924 10:02:17.283262 21218 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:17.283384 20104 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:40203
I20260924 10:02:17.283679 21218 heartbeater.cc:511] Master 127.13.162.60:33795 requested a full tablet report, sending...
I20260924 10:02:17.284809 20890 ts_manager.cc:194] Registered new tserver with Master: 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257)
I20260924 10:02:17.286819 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.234013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.13.162.3:33439)
0924 10:02:17.262608 (+ 28595us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.262616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.262651 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:02:17.262720 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.262728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.262997 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.263166 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.266229 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.268217 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.269904 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.270807 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.270924 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.279320 (+ 8396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.279339 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.279357 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.279400 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.283280 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.284352 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.284360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.284362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.284490 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.284863 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.284869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.284872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.285274 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.285540 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.286094 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.286459 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28444,"thread_start_us":28306,"threads_started":1}
I20260924 10:02:17.288090 20556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:17.370527 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20433
I20260924 10:02:17.383291 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:33439
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:44465
--webserver_port=41135
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:02:17.677590 21068 heartbeater.cc:503] Master 127.13.162.62:44465 was elected leader, sending a full tablet report...
W20260924 10:02:17.705133 21231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:17.705466 21231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:17.705611 21231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:17.713114 21231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:17.713248 21231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:17.713414 21231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:17.723237 21231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:33439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=41135
--tserver_master_addrs=127.13.162.62:44465,127.13.162.61:45497,127.13.162.60:33795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:17.724597 21231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:17.725898 21231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:17.738817 21239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:17.738845 21237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:17.739260 21236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:17.739749 21231 server_base.cc:1034] running on GCE node
I20260924 10:02:17.740339 21231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:17.741465 21231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:17.742671 21231 hybrid_clock.cc:648] HybridClock initialized: now 1790244137742605 us; error 49 us; skew 500 ppm
I20260924 10:02:17.742982 21231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:17.746294 21231 init.cc:377] Logged in from keytab as oryx/127.13.162.3@KRBTEST.COM (short username oryx)
I20260924 10:02:17.747699 21231 webserver.cc:460] Webserver started at http://127.13.162.3:41135/ using document root <none> and password file <none>
I20260924 10:02:17.748394 21231 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:17.748517 21231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:17.753908 21231 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:17.756731 21246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:17.758028 21231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:02:17.758210 21231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "f86c2f01f67b41d09e1f94f36c9872cd"
format_stamp: "Formatted at 2026-09-24 10:02:14 on dist-test-slave-jvsj"
server_key: "ecd1ef9fc0179c8a1c248e98e9580f1b"
server_key_iv: "9a6f797304b7054c7a56528b92795178"
server_key_version: "encryptionkey@0"
I20260924 10:02:17.759176 21231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:17.770813 21231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:17.774964 21231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:17.775235 21231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:17.776152 21231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:17.777455 21231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:17.777539 21231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:17.777635 21231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:17.777706 21231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:17.805554 21231 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:33439
I20260924 10:02:17.805620 21358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:33439 every 8 connection(s)
I20260924 10:02:17.806959 21231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:17.812844 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 21231
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.60@KRBTEST.COM
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.61@KRBTEST.COM
Sep 24 10:02:17 dist-test-slave-jvsj krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244137, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:17.863806 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.810305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:39287 (local address 127.13.162.61:45497)
0924 10:02:17.842730 (+ 32425us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.842737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.844438 (+ 1701us) server_negotiation.cc:407] Connection header received
0924 10:02:17.844605 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.844611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.844688 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.844789 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.846023 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.847026 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.849369 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.850008 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.850140 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.856232 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.856238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.856242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.856292 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.859545 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.860568 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.860574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.860577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.860746 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.861346 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.861351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.861355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.861701 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.861827 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.863068 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.863384 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32252,"thread_start_us":32124,"threads_started":1}
I20260924 10:02:17.865782 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.809759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:50455 (local address 127.13.162.62:44465)
0924 10:02:17.846307 (+ 36548us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.846330 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.846348 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:17.846401 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.846406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.846478 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.846581 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.847512 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.848577 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.851159 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.851752 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.851876 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.858524 (+ 6648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.858532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.858537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.858581 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.861967 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.862833 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.862838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.862841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.863028 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.864003 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.864105 (+ 102us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.864182 (+ 77us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.864530 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.864695 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.865220 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.865434 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36431,"thread_start_us":36331,"threads_started":1}
I20260924 10:02:17.866243 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.810415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.13.162.3:39287)
0924 10:02:17.844340 (+ 33925us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.844369 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.844488 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.844956 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.844961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.845549 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:17.845883 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.845895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.847508 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.847512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.849226 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.849241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.850467 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.850473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.850676 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.852082 (+ 1406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:17.852104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:17.855979 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:17.859731 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.859742 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.859764 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.860424 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.860910 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.860914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.860918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.861147 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.861938 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:17.861946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:17.862618 (+ 672us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.864159 (+ 1541us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.864746 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33763,"mutex_wait_us":343,"thread_start_us":25167,"threads_started":1}
I20260924 10:02:17.866639 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.810415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:50455)
0924 10:02:17.832279 (+ 21864us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.832365 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.835416 (+ 3051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.846747 (+ 11331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.846751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.847179 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:17.847370 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.847377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.849396 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.849400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.850933 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.850949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.851976 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.851983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.852213 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.852784 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:17.852809 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:17.855996 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:17.862087 (+ 6091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.862092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.862096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.862637 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.863154 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.863159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.863163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.863359 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.864648 (+ 1289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:17.864659 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:17.864813 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.864946 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.865252 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21379,"thread_start_us":33482,"threads_started":1}
I20260924 10:02:17.868497 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.810546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.13.162.3:53055)
0924 10:02:17.837715 (+ 27169us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.837741 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.837892 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.839339 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.839354 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.840433 (+ 1079us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:17.841458 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.841485 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.843691 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.843698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.846304 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.846350 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.847937 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.847945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.848498 (+ 553us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.849401 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:17.849434 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:17.853487 (+ 4053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:17.863241 (+ 9754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.863245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.863248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.863754 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.866425 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:17.866429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:17.866433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:17.866990 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:17.867615 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:17.867618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:17.867742 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.867852 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.868109 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26864,"spinlock_wait_cycles":25856,"thread_start_us":33074,"threads_started":1}
I20260924 10:02:17.868765 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.809978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:53055 (local address 127.13.162.60:33795)
0924 10:02:17.838428 (+ 28450us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.838436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.838458 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:17.838523 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.838529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.838696 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.838811 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.841718 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.843469 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.846533 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.847287 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.847398 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.853813 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.853821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.853826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.853867 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.858282 (+ 4415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.864826 (+ 6544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.864832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.864835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.865008 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.867119 (+ 2111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.867125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.867129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.867439 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.867543 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.867992 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.868221 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28291,"thread_start_us":28116,"threads_started":1}
I20260924 10:02:17.870463 21359 heartbeater.cc:347] Connected to a master server at 127.13.162.60:33795
I20260924 10:02:17.870833 21359 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:17.871745 21359 heartbeater.cc:511] Master 127.13.162.60:33795 requested a full tablet report, sending...
I20260924 10:02:17.873766 20890 ts_manager.cc:194] Registered new tserver with Master: f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439)
I20260924 10:02:17.878984 21360 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44465
I20260924 10:02:17.879115 21360 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:17.879555 21360 heartbeater.cc:511] Master 127.13.162.62:44465 requested a full tablet report, sending...
I20260924 10:02:17.880429 21361 heartbeater.cc:347] Connected to a master server at 127.13.162.61:45497
I20260924 10:02:17.880555 21361 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:17.880964 21361 heartbeater.cc:511] Master 127.13.162.61:45497 requested a full tablet report, sending...
I20260924 10:02:17.881770 20104 ts_manager.cc:194] Re-registered known tserver with Master: f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439)
I20260924 10:02:17.882544 20702 ts_manager.cc:194] Registered new tserver with Master: f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439)
I20260924 10:02:17.882872 20104 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:50455
I20260924 10:02:17.888839 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.856169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.13.162.60:33795)
0924 10:02:17.869551 (+ 13382us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.869557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.871305 (+ 1748us) server_negotiation.cc:407] Connection header received
0924 10:02:17.871472 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.871477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.871563 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.871650 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.873392 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.874573 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.876386 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.876993 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.877084 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.882175 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.882182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.882187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.882222 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.885815 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.886613 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.886619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.886623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.886780 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.887157 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.887162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.887165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.887469 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.887714 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.888270 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.888479 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13277,"thread_start_us":24035,"threads_started":1}
I20260924 10:02:17.891222 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.856440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.13.162.62:44465)
0924 10:02:17.865999 (+ 9559us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.866005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.866022 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:02:17.866067 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.866072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.866138 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.866236 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.875476 (+ 9240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.876494 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.879319 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.879893 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.879983 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.884269 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.884275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.884280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.884314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.887209 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.887950 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.887955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.887958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.888093 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.889863 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.889868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.889870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.890151 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.890405 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.890738 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.890970 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9468,"thread_start_us":21755,"threads_started":1}
I20260924 10:02:17.896317 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.862928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.13.162.61:45497)
0924 10:02:17.864046 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.864051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.879866 (+ 15815us) server_negotiation.cc:407] Connection header received
0924 10:02:17.879931 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.879935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.880029 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.880117 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:17.885333 (+ 5216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.886393 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.888132 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.888725 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.888848 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.891450 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:17.891455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:17.891460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:17.891495 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:17.893949 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.894634 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.894638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.894640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.894769 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:17.895117 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:17.895122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:17.895124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:17.895380 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:17.895574 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.895887 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.896055 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":16795,"threads_started":1}
I20260924 10:02:17.895634 20104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:17.902802 20104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:17.945310 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.920646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.13.162.2:44257)
0924 10:02:17.926717 (+ 6071us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.926728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.937516 (+ 10788us) server_negotiation.cc:407] Connection header received
0924 10:02:17.937824 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.937834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.938146 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.938389 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:17.938884 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.940511 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.943440 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.944368 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.944472 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.944667 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.944966 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.945115 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5918,"thread_start_us":5832,"threads_started":1}
I20260924 10:02:17.945353 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.920351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44257 (local address 127.0.0.1:60176)
0924 10:02:17.937386 (+ 17035us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.937409 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.937537 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.938360 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.938364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.938388 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:17.938738 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.938749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.940659 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.940665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.942996 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.943012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.944569 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.944576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.944776 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.944812 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.944891 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.945033 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12343,"thread_start_us":21119,"threads_started":1}
I20260924 10:02:17.948961 21153 tablet_service.cc:1467] Processing CreateTablet for tablet 837e41adf70d4f8593acf0a462a83428 (DEFAULT_TABLE table=test-table [id=d90de73246eb4bc69ce21a1dd998d670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:17.950505 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.932939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:33439 (local address 127.0.0.1:44260)
0924 10:02:17.942023 (+ 9084us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.942044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.942236 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.942770 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.942774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.942798 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:17.943246 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.943258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.945589 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.945594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.947889 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.947903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.949627 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.949634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.949900 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.949934 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.950028 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.950156 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8963,"mutex_wait_us":79,"spinlock_wait_cycles":896,"thread_start_us":8817,"threads_started":1}
I20260924 10:02:17.950541 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e41adf70d4f8593acf0a462a83428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:17.950587 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.933410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.13.162.3:33439)
0924 10:02:17.937136 (+ 3726us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.937142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.942124 (+ 4982us) server_negotiation.cc:407] Connection header received
0924 10:02:17.942426 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.942432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.942607 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.942786 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:17.943423 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.944512 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.948045 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.949439 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.949564 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.949775 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.950068 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.950229 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3573,"spinlock_wait_cycles":62720,"thread_start_us":3418,"threads_started":1}
I20260924 10:02:17.951612 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.920902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:45073 (local address 127.0.0.1:43232)
0924 10:02:17.941249 (+ 20347us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:17.941267 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:17.941382 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:17.942034 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:17.942039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:17.942065 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:17.942432 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.942443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.945937 (+ 3494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.945942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.948122 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:17.948132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.950772 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.950781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:17.950991 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.951024 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:02:17.951121 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:02:17.951250 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20245,"mutex_wait_us":19,"spinlock_wait_cycles":25856,"thread_start_us":11665,"threads_started":1}
I20260924 10:02:17.951668 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:17.921105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.13.162.1:45073)
0924 10:02:17.935516 (+ 14411us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:17.935523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:17.941367 (+ 5844us) server_negotiation.cc:407] Connection header received
0924 10:02:17.941531 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:17.941541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:17.941824 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:17.942034 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:17.943060 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.945807 (+ 2747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.948250 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:17.949800 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:17.949931 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:17.950125 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:17.951204 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0924 10:02:17.951372 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14270,"thread_start_us":14398,"threads_started":1}
I20260924 10:02:17.953909 21294 tablet_service.cc:1467] Processing CreateTablet for tablet 837e41adf70d4f8593acf0a462a83428 (DEFAULT_TABLE table=test-table [id=d90de73246eb4bc69ce21a1dd998d670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:17.955338 21294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e41adf70d4f8593acf0a462a83428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:17.957281 21002 tablet_service.cc:1467] Processing CreateTablet for tablet 837e41adf70d4f8593acf0a462a83428 (DEFAULT_TABLE table=test-table [id=d90de73246eb4bc69ce21a1dd998d670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:17.958541 21002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e41adf70d4f8593acf0a462a83428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:17.968899 21393 tablet_bootstrap.cc:492] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Bootstrap starting.
I20260924 10:02:17.969846 21395 tablet_bootstrap.cc:492] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Bootstrap starting.
I20260924 10:02:17.972873 21395 tablet_bootstrap.cc:654] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:17.973299 21393 tablet_bootstrap.cc:654] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:17.973766 21394 tablet_bootstrap.cc:492] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Bootstrap starting.
I20260924 10:02:17.974411 21395 log.cc:824] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:17.974445 21393 log.cc:824] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:17.977020 21395 tablet_bootstrap.cc:492] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: No bootstrap required, opened a new log
I20260924 10:02:17.977130 21394 tablet_bootstrap.cc:654] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:17.977285 21395 ts_tablet_manager.cc:1397] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 10:02:17.977334 21393 tablet_bootstrap.cc:492] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: No bootstrap required, opened a new log
I20260924 10:02:17.977579 21393 ts_tablet_manager.cc:1397] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:02:17.978276 21394 log.cc:824] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:17.981446 21394 tablet_bootstrap.cc:492] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: No bootstrap required, opened a new log
I20260924 10:02:17.981691 21394 ts_tablet_manager.cc:1397] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:02:17.984937 21395 raft_consensus.cc:348] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:17.985277 21395 raft_consensus.cc:374] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:17.985369 21395 raft_consensus.cc:729] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86c2f01f67b41d09e1f94f36c9872cd, State: Initialized, Role: FOLLOWER
I20260924 10:02:17.985919 21395 consensus_queue.cc:260] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:17.987461 21395 ts_tablet_manager.cc:1428] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260924 10:02:17.987196 21393 raft_consensus.cc:348] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:17.987596 21393 raft_consensus.cc:374] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:17.987705 21393 raft_consensus.cc:729] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82add254d9d8479480e7018a651d2291, State: Initialized, Role: FOLLOWER
I20260924 10:02:17.988404 21360 heartbeater.cc:503] Master 127.13.162.62:44465 was elected leader, sending a full tablet report...
I20260924 10:02:17.988272 21393 consensus_queue.cc:260] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:17.994665 21219 heartbeater.cc:503] Master 127.13.162.62:44465 was elected leader, sending a full tablet report...
I20260924 10:02:17.995393 21393 ts_tablet_manager.cc:1428] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
I20260924 10:02:17.995223 21394 raft_consensus.cc:348] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:17.995675 21394 raft_consensus.cc:374] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:17.995781 21394 raft_consensus.cc:729] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa49ed9187a42568c7ff491e12fb40b, State: Initialized, Role: FOLLOWER
I20260924 10:02:17.996387 21394 consensus_queue.cc:260] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:17.999926 21394 ts_tablet_manager.cc:1428] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260924 10:02:18.052601 21400 raft_consensus.cc:482] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:18.052915 21400 raft_consensus.cc:504] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:18.054531 21400 leader_election.cc:288] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073), f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439)
W20260924 10:02:18.059777 21362 tablet.cc:2369] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:02:18.061022 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.055781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:41155 (local address 127.13.162.3:33439)
0924 10:02:18.055965 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.055970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.056219 (+ 249us) server_negotiation.cc:407] Connection header received
0924 10:02:18.056386 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.056390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.056444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.056512 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:18.056952 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.057635 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.059336 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.060198 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.060348 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.060417 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.060684 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.060805 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:02:18.061265 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.054950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:37629 (local address 127.13.162.1:45073)
0924 10:02:18.055125 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.055129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.055660 (+ 531us) server_negotiation.cc:407] Connection header received
0924 10:02:18.055909 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.055916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.055989 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.056048 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:18.056663 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.057361 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.059168 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.060204 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.060345 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.060415 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.060897 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.061021 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:02:18.061368 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.054875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:45073 (local address 127.13.162.2:37629)
0924 10:02:18.055570 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:18.055587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:18.055721 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:18.056214 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:18.056217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:18.056273 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:18.056530 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:18.056538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.057490 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.057495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:18.059023 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:18.059035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.060562 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.060569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:18.060752 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.060776 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:02:18.060853 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:02:18.060975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":49,"spinlock_wait_cycles":4480,"thread_start_us":381,"threads_started":1}
I20260924 10:02:18.061412 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.055639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:33439 (local address 127.13.162.2:41155)
0924 10:02:18.056122 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:18.056135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:18.056275 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:18.056540 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:18.056545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:18.056567 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:18.056826 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:18.056834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.057765 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.057770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:18.059190 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:18.059202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.060382 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.060388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:18.060537 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.060563 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:02:18.060643 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:02:18.060783 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":2,"spinlock_wait_cycles":8960,"thread_start_us":274,"threads_started":1}
I20260924 10:02:18.062260 21314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e41adf70d4f8593acf0a462a83428" candidate_uuid: "82add254d9d8479480e7018a651d2291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" is_pre_election: true
I20260924 10:02:18.062269 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e41adf70d4f8593acf0a462a83428" candidate_uuid: "82add254d9d8479480e7018a651d2291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa49ed9187a42568c7ff491e12fb40b" is_pre_election: true
I20260924 10:02:18.062853 21314 raft_consensus.cc:2454] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82add254d9d8479480e7018a651d2291 in term 0.
I20260924 10:02:18.062858 21022 raft_consensus.cc:2454] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82add254d9d8479480e7018a651d2291 in term 0.
I20260924 10:02:18.063476 21109 leader_election.cc:302] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82add254d9d8479480e7018a651d2291, f86c2f01f67b41d09e1f94f36c9872cd; no voters:
I20260924 10:02:18.063872 21400 raft_consensus.cc:2781] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:18.064004 21400 raft_consensus.cc:482] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:18.064100 21400 raft_consensus.cc:3037] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:18.066252 21400 raft_consensus.cc:504] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:18.066846 21400 leader_election.cc:288] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [CANDIDATE]: Term 1 election: Requested vote from peers 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073), f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439)
I20260924 10:02:18.067227 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e41adf70d4f8593acf0a462a83428" candidate_uuid: "82add254d9d8479480e7018a651d2291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa49ed9187a42568c7ff491e12fb40b"
I20260924 10:02:18.067296 21314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e41adf70d4f8593acf0a462a83428" candidate_uuid: "82add254d9d8479480e7018a651d2291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86c2f01f67b41d09e1f94f36c9872cd"
I20260924 10:02:18.067394 21022 raft_consensus.cc:3037] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:18.067476 21314 raft_consensus.cc:3037] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:18.069464 21314 raft_consensus.cc:2454] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82add254d9d8479480e7018a651d2291 in term 1.
I20260924 10:02:18.069471 21022 raft_consensus.cc:2454] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82add254d9d8479480e7018a651d2291 in term 1.
I20260924 10:02:18.069794 21109 leader_election.cc:302] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82add254d9d8479480e7018a651d2291, 9fa49ed9187a42568c7ff491e12fb40b; no voters:
I20260924 10:02:18.070039 21400 raft_consensus.cc:2781] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:18.070511 21400 raft_consensus.cc:686] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 1 LEADER]: Becoming Leader. State: Replica: 82add254d9d8479480e7018a651d2291, State: Running, Role: LEADER
I20260924 10:02:18.071033 21400 consensus_queue.cc:237] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } }
I20260924 10:02:18.075912 20105 catalog_manager.cc:5495] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82add254d9d8479480e7018a651d2291 (127.13.162.2). New cstate: current_term: 1 leader_uuid: "82add254d9d8479480e7018a651d2291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } health_report { overall_health: UNKNOWN } } }
I20260924 10:02:18.093528 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.088643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.13.162.2:44257)
0924 10:02:18.088803 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.088808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.088824 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:02:18.088979 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.088982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.089054 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.089204 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:18.089472 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.090150 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.091494 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.091928 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.092030 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.092459 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:18.092993 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:18.093105 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.093246 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.093309 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:18.101387 21022 raft_consensus.cc:1261] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Refusing update from remote peer 82add254d9d8479480e7018a651d2291: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:18.101385 21314 raft_consensus.cc:1261] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Refusing update from remote peer 82add254d9d8479480e7018a651d2291: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:18.102360 21404 consensus_queue.cc:1035] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:18.102737 21400 consensus_queue.cc:1035] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:02:18.107044 21070 tablet.cc:2369] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:02:18.136234 20105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:18.136545 20105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:18.144927 20105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 643687b7468c438ebc39c28da7f05a82: Sending DeleteTablet for 3 replicas of tablet 837e41adf70d4f8593acf0a462a83428
I20260924 10:02:18.146037 21002 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e41adf70d4f8593acf0a462a83428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:18 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:43232
I20260924 10:02:18.146073 21153 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e41adf70d4f8593acf0a462a83428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:18 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:60176
I20260924 10:02:18.147464 21294 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e41adf70d4f8593acf0a462a83428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:18 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:44260
I20260924 10:02:18.158299 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20073
I20260924 10:02:18.162529 20135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:18.167460 21418 tablet_replica.cc:330] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: stopping tablet replica
I20260924 10:02:18.167937 21418 raft_consensus.cc:2229] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:18.168474 21418 raft_consensus.cc:2258] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:18.174829 21418 ts_tablet_manager.cc:1905] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:18.181903 21420 tablet_replica.cc:330] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: stopping tablet replica
I20260924 10:02:18.183264 21420 raft_consensus.cc:2229] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:18.184302 21419 tablet_replica.cc:330] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: stopping tablet replica
I20260924 10:02:18.184738 21419 raft_consensus.cc:2229] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:18.186086 21420 raft_consensus.cc:2258] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:18.187973 21418 ts_tablet_manager.cc:1918] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:18.188062 21419 raft_consensus.cc:2258] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:18.188164 21418 log.cc:1196] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/837e41adf70d4f8593acf0a462a83428
I20260924 10:02:18.188495 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.149265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.13.162.61:45497)
0924 10:02:18.149433 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.149439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.157195 (+ 7756us) server_negotiation.cc:407] Connection header received
0924 10:02:18.159068 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.159074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.159152 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.159216 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:18.163345 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.164567 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.168932 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.169577 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.169672 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.180201 (+ 10529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:18.180210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:18.180215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:18.180276 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:18.184385 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.185393 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.185399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.185403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.185580 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.186087 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.186092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.186096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.186736 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:18.186979 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.187688 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.187944 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:18.188858 21418 ts_tablet_manager.cc:1939] T 837e41adf70d4f8593acf0a462a83428 P 9fa49ed9187a42568c7ff491e12fb40b: Deleting consensus metadata
I20260924 10:02:18.191692 21420 ts_tablet_manager.cc:1905] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:18.199788 21419 ts_tablet_manager.cc:1905] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:18.199934 21420 ts_tablet_manager.cc:1918] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:18.200071 21420 log.cc:1196] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/837e41adf70d4f8593acf0a462a83428
I20260924 10:02:18.200551 21420 ts_tablet_manager.cc:1939] T 837e41adf70d4f8593acf0a462a83428 P f86c2f01f67b41d09e1f94f36c9872cd: Deleting consensus metadata
I20260924 10:02:18.206537 21419 ts_tablet_manager.cc:1918] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:18.206703 21419 log.cc:1196] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/837e41adf70d4f8593acf0a462a83428
I20260924 10:02:18.206857 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.157339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.13.162.60:33795)
0924 10:02:18.157485 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.157491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.189858 (+ 32367us) server_negotiation.cc:407] Connection header received
0924 10:02:18.190110 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.190116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.190211 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.190298 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:18.192353 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.193541 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.195431 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.196072 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.196186 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.199775 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:18.199784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:18.199788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:18.199828 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:18.203470 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.204298 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.204304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.204308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.204465 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.204901 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.204906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.204910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.205305 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:18.205576 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.206090 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.206340 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:02:18.207242 21419 ts_tablet_manager.cc:1939] T 837e41adf70d4f8593acf0a462a83428 P 82add254d9d8479480e7018a651d2291: Deleting consensus metadata
I20260924 10:02:18.333298 20089 catalog_manager.cc:4841] TS 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257): tablet 837e41adf70d4f8593acf0a462a83428 (table test-table [id=d90de73246eb4bc69ce21a1dd998d670]) successfully deleted
I20260924 10:02:18.333422 20092 catalog_manager.cc:4841] TS f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439): tablet 837e41adf70d4f8593acf0a462a83428 (table test-table [id=d90de73246eb4bc69ce21a1dd998d670]) successfully deleted
I20260924 10:02:18.333737 20092 catalog_manager.cc:4841] TS 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073): tablet 837e41adf70d4f8593acf0a462a83428 (table test-table [id=d90de73246eb4bc69ce21a1dd998d670]) successfully deleted
I20260924 10:02:18.334206 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20073
I20260924 10:02:18.376879 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.358020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.13.162.61:45497)
0924 10:02:18.358223 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.358230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.359255 (+ 1025us) server_negotiation.cc:407] Connection header received
0924 10:02:18.359525 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.359532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.359617 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.359708 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:18.362203 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.363571 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.365786 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.366592 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.366723 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.370399 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:18.370406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:18.370411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:18.370454 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:18.373838 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.374551 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.374558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.374562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.374702 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.375146 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.375150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.375154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.375492 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:18.375723 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.376143 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.376542 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:02:18.382277 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:18.357657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.13.162.60:33795)
0924 10:02:18.358204 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:18.358211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:18.365871 (+ 7660us) server_negotiation.cc:407] Connection header received
0924 10:02:18.366408 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:18.366415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:18.366503 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:18.366578 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:18.369407 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.370493 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.371816 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:18.372265 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:18.372393 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:18.376653 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:18.376658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:18.376662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:18.376694 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:18.379763 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.380335 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.380339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.380342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.380459 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:18.380796 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:18.380801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:18.380805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:18.381112 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:18.381335 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:18.381706 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 10:02:18.381929 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260924 10:02:19.124907 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.124100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:37601)
0924 10:02:19.124559 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.124705 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":259,"threads_started":1}
W20260924 10:02:19.125378 21219 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:44465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.126699 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.125692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:35609)
0924 10:02:19.126382 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.126510 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":478,"threads_started":1}
W20260924 10:02:19.127094 21068 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:44465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.128362 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.127421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:35609)
0924 10:02:19.128020 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.128153 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":385,"threads_started":1}
W20260924 10:02:19.128682 21360 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:44465 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.644593 21433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 643687b7468c438ebc39c28da7f05a82)
I20260924 10:02:19.644866 21433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } }
I20260924 10:02:19.645215 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 643687b7468c438ebc39c28da7f05a82)
I20260924 10:02:19.645440 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } }
I20260924 10:02:19.646363 21433 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465), e6cad33b49a1472489cdfbc23b253451 (127.13.162.60:33795)
I20260924 10:02:19.647697 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.646496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37682)
0924 10:02:19.647140 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.647390 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":470,"threads_started":1}
W20260924 10:02:19.648051 20686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.659832 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.646822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37694)
0924 10:02:19.659284 (+ 12462us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.659539 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12348,"threads_started":1}
I20260924 10:02:19.660835 21434 leader_election.cc:288] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465), 4b7af120325d4809b3aff9cd14e1026c (127.13.162.61:45497)
W20260924 10:02:19.663213 20874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.664901 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.659169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37698)
0924 10:02:19.664514 (+ 5345us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.664695 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5228}
W20260924 10:02:19.665169 20686 leader_election.cc:334] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465): Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.665702 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.665351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37714)
0924 10:02:19.665534 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.665601 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 10:02:19.665866 20874 leader_election.cc:334] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465): Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.665962 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.646619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:33795 (local address 127.0.0.1:50838)
0924 10:02:19.653347 (+ 6728us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.653369 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.653513 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.658771 (+ 5258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.658776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.658817 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.659073 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.659087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.660763 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.660768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.663010 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.663073 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.665173 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.665178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.665373 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.665413 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.665509 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.665637 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6615,"mutex_wait_us":3,"thread_start_us":17656,"threads_started":1}
I20260924 10:02:19.666641 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.646880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.13.162.60:33795)
0924 10:02:19.658353 (+ 11473us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.658359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.658379 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:02:19.658449 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.658453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.658541 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.659216 (+ 675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.659448 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.660609 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.663250 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.665009 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.666122 (+ 1113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.666205 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.666242 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.666355 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":409,"server-negotiator.queue_time_us":11329,"thread_start_us":11745,"threads_started":1}
I20260924 10:02:19.667141 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7af120325d4809b3aff9cd14e1026c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e6cad33b49a1472489cdfbc23b253451" is_pre_election: true
I20260924 10:02:19.667380 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.653277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:45497 (local address 127.0.0.1:40828)
0924 10:02:19.659694 (+ 6417us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.659711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.659839 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.660367 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.660372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.660409 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.660727 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.660737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.662424 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.662429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.664979 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.664997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.666646 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.666652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.666835 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.666866 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.666953 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.667056 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6287,"mutex_wait_us":297,"thread_start_us":5773,"threads_started":1}
I20260924 10:02:19.667426 20900 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7af120325d4809b3aff9cd14e1026c in term 1.
I20260924 10:02:19.667807 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.653251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.13.162.61:45497)
0924 10:02:19.657037 (+ 3786us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.657043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.659939 (+ 2896us) server_negotiation.cc:407] Connection header received
0924 10:02:19.659996 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.660000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.660061 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.660139 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.660923 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.662192 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.665166 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.666450 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.666556 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.666652 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.667394 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.667518 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":3647,"thread_start_us":3443,"threads_started":1}
I20260924 10:02:19.667877 20687 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b7af120325d4809b3aff9cd14e1026c, e6cad33b49a1472489cdfbc23b253451; no voters: 643687b7468c438ebc39c28da7f05a82
I20260924 10:02:19.668260 21433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:02:19.668375 21433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 643687b7468c438ebc39c28da7f05a82)
I20260924 10:02:19.668488 21433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:02:19.668382 20712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6cad33b49a1472489cdfbc23b253451" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b7af120325d4809b3aff9cd14e1026c" is_pre_election: true
I20260924 10:02:19.670475 21433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } }
I20260924 10:02:19.670900 20712 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6cad33b49a1472489cdfbc23b253451 in current term 2: Already voted for candidate 4b7af120325d4809b3aff9cd14e1026c in this term.
I20260924 10:02:19.671069 21433 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [CANDIDATE]: Term 2 election: Requested vote from peers 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465), e6cad33b49a1472489cdfbc23b253451 (127.13.162.60:33795)
I20260924 10:02:19.671515 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.671137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37718)
0924 10:02:19.671286 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.671370 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:02:19.671548 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7af120325d4809b3aff9cd14e1026c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e6cad33b49a1472489cdfbc23b253451"
I20260924 10:02:19.671729 20900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:02:19.672289 20876 leader_election.cc:302] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6cad33b49a1472489cdfbc23b253451; no voters: 4b7af120325d4809b3aff9cd14e1026c, 643687b7468c438ebc39c28da7f05a82
I20260924 10:02:19.672405 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.672109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37726)
0924 10:02:19.672250 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.672320 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:02:19.672578 20686 leader_election.cc:334] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465): Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
I20260924 10:02:19.674077 20900 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7af120325d4809b3aff9cd14e1026c in term 2.
I20260924 10:02:19.674252 21434 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:02:19.674603 20687 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b7af120325d4809b3aff9cd14e1026c, e6cad33b49a1472489cdfbc23b253451; no voters: 643687b7468c438ebc39c28da7f05a82
I20260924 10:02:19.674845 21433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:02:19.675235 21433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [term 2 LEADER]: Becoming Leader. State: Replica: 4b7af120325d4809b3aff9cd14e1026c, State: Running, Role: LEADER
I20260924 10:02:19.675577 21433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } }
I20260924 10:02:19.677325 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7af120325d4809b3aff9cd14e1026c. Latest consensus state: current_term: 2 leader_uuid: "4b7af120325d4809b3aff9cd14e1026c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:19.677680 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:19.678287 21445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:19.681578 21445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:19.682360 21445 catalog_manager.cc:1259] Loaded cluster ID: e2e4455f26a84d5aabfb9cb777021f6b
I20260924 10:02:19.682437 21445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:19.683490 21445 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:19.684260 21445 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Loaded TSK: 0
I20260924 10:02:19.685052 21445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:02:19.717664 20701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:19.724488 20701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:19.728510 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.728127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37738)
0924 10:02:19.728262 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.728334 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 10:02:19.728715 20900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [term 2 FOLLOWER]: Refusing update from remote peer 4b7af120325d4809b3aff9cd14e1026c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 10:02:19.729393 21443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 10:02:19.729528 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.729199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37752)
0924 10:02:19.729333 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.729400 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:02:19.729688 20686 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c -> Peer 643687b7468c438ebc39c28da7f05a82 (127.13.162.62:44465): Couldn't send request to peer 643687b7468c438ebc39c28da7f05a82. Status: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:02:19.732041 21434 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7af120325d4809b3aff9cd14e1026c. Latest consensus state: current_term: 2 leader_uuid: "4b7af120325d4809b3aff9cd14e1026c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:19.732332 21434 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:19.733752 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4b7af120325d4809b3aff9cd14e1026c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:19.734380 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:19.734452 21434 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4b7af120325d4809b3aff9cd14e1026c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "643687b7468c438ebc39c28da7f05a82" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44465 } } peers { permanent_uuid: "4b7af120325d4809b3aff9cd14e1026c" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 45497 } attrs { promote: false } } peers { permanent_uuid: "e6cad33b49a1472489cdfbc23b253451" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 33795 } attrs { promote: false } } }
I20260924 10:02:19.734691 21434 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cad33b49a1472489cdfbc23b253451 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:02:19.734885 21447 mvcc.cc:205] Tried to move back new op lower bound from 7332839996322013184 to 7332839996114071552. Current Snapshot: MvccSnapshot[applied={T|T < 7332839996322013184 or (T in {7332839996322013184})}]
I20260924 10:02:19.747694 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.742437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.13.162.2:44257)
0924 10:02:19.742843 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.742848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.742863 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:02:19.742914 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.742917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.742977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.743062 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.743517 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.744182 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.746012 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.746805 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.746916 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.747016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.747285 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.747411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":206,"threads_started":1}
I20260924 10:02:19.747700 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.742069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44257 (local address 127.0.0.1:60188)
0924 10:02:19.742584 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.742608 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.742718 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.743085 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.743089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.743110 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.743390 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.743399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.744305 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.744311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.745869 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.745883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.746986 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.746991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.747149 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.747172 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.747249 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.747374 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260924 10:02:19.748654 21153 tablet_service.cc:1467] Processing CreateTablet for tablet 13237b3d2a964dbcb6e25bd99f0d59fe (DEFAULT_TABLE table=test-table [id=4129f8ba975b48a4972a23a77eb757d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:19.749115 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13237b3d2a964dbcb6e25bd99f0d59fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:19.754694 21456 tablet_bootstrap.cc:492] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Bootstrap starting.
I20260924 10:02:19.756475 21456 tablet_bootstrap.cc:654] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:19.756938 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.741918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:33439 (local address 127.0.0.1:44268)
0924 10:02:19.742040 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.742056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.742144 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.750986 (+ 8842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.750992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.751022 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.751284 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.751295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.752440 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.752445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.754180 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.754193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.756223 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.756229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.756374 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.756396 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.756478 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.756618 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 10:02:19.757189 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.741561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:45073 (local address 127.0.0.1:43246)
0924 10:02:19.741725 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.741743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.741837 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.751714 (+ 9877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.751719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.751744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.751873 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.751882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.753224 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.753229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.754975 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.754988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.756513 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.756518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.756683 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.756713 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.756803 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.756914 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 10:02:19.757278 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.742152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.13.162.1:45073)
0924 10:02:19.751410 (+ 9258us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.751416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.751434 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:19.751496 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.751501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.751593 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.751683 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.752005 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.753074 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.755177 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.756352 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.756479 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.756588 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.756861 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.756986 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9144,"thread_start_us":9018,"threads_started":1}
I20260924 10:02:19.757736 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.742551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.13.162.3:33439)
0924 10:02:19.750664 (+ 8113us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.750671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.750690 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:02:19.750752 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.750757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.750835 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.750935 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.751426 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.752298 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.755200 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.756034 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.756143 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.756241 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.757319 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.757464 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7998,"thread_start_us":7905,"threads_started":1}
I20260924 10:02:19.758180 21002 tablet_service.cc:1467] Processing CreateTablet for tablet 13237b3d2a964dbcb6e25bd99f0d59fe (DEFAULT_TABLE table=test-table [id=4129f8ba975b48a4972a23a77eb757d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:19.758378 21294 tablet_service.cc:1467] Processing CreateTablet for tablet 13237b3d2a964dbcb6e25bd99f0d59fe (DEFAULT_TABLE table=test-table [id=4129f8ba975b48a4972a23a77eb757d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:19.758605 21002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13237b3d2a964dbcb6e25bd99f0d59fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:19.758746 21294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13237b3d2a964dbcb6e25bd99f0d59fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:19.764811 21458 tablet_bootstrap.cc:492] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Bootstrap starting.
I20260924 10:02:19.765372 21456 tablet_bootstrap.cc:492] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: No bootstrap required, opened a new log
I20260924 10:02:19.766129 21456 ts_tablet_manager.cc:1397] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20260924 10:02:19.766554 21458 tablet_bootstrap.cc:654] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:19.767006 21456 raft_consensus.cc:348] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.767557 21456 raft_consensus.cc:374] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:19.767627 21459 tablet_bootstrap.cc:492] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Bootstrap starting.
I20260924 10:02:19.767665 21456 raft_consensus.cc:729] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82add254d9d8479480e7018a651d2291, State: Initialized, Role: FOLLOWER
I20260924 10:02:19.767874 21456 consensus_queue.cc:260] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.768503 21456 ts_tablet_manager.cc:1428] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:02:19.769219 21459 tablet_bootstrap.cc:654] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:19.771237 21458 tablet_bootstrap.cc:492] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: No bootstrap required, opened a new log
I20260924 10:02:19.771437 21458 ts_tablet_manager.cc:1397] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 10:02:19.772416 21458 raft_consensus.cc:348] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.772697 21458 raft_consensus.cc:374] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:19.772796 21458 raft_consensus.cc:729] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa49ed9187a42568c7ff491e12fb40b, State: Initialized, Role: FOLLOWER
I20260924 10:02:19.773059 21458 consensus_queue.cc:260] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.773855 21458 ts_tablet_manager.cc:1428] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:02:19.774809 21220 heartbeater.cc:503] Master 127.13.162.61:45497 was elected leader, sending a full tablet report...
I20260924 10:02:19.774972 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.774038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:34347)
0924 10:02:19.774588 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.774741 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"threads_started":1}
I20260924 10:02:19.777283 21069 heartbeater.cc:503] Master 127.13.162.61:45497 was elected leader, sending a full tablet report...
I20260924 10:02:19.777829 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.777406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:36583)
0924 10:02:19.777620 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.777699 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 10:02:19.778795 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.768904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:36795)
0924 10:02:19.778533 (+ 9629us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.778634 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9495,"thread_start_us":9364,"threads_started":1}
I20260924 10:02:19.778825 21459 tablet_bootstrap.cc:492] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: No bootstrap required, opened a new log
I20260924 10:02:19.778992 21459 ts_tablet_manager.cc:1397] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20260924 10:02:19.779718 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.779419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:48591)
0924 10:02:19.779551 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.779614 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:02:19.779747 21459 raft_consensus.cc:348] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.779973 21459 raft_consensus.cc:374] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:19.780063 21459 raft_consensus.cc:729] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86c2f01f67b41d09e1f94f36c9872cd, State: Initialized, Role: FOLLOWER
I20260924 10:02:19.780263 21459 consensus_queue.cc:260] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.780923 21459 ts_tablet_manager.cc:1428] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:02:19.782037 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.781192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:43377)
0924 10:02:19.781654 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.781785 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":642,"threads_started":1}
I20260924 10:02:19.783176 21361 heartbeater.cc:503] Master 127.13.162.61:45497 was elected leader, sending a full tablet report...
I20260924 10:02:19.783445 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.782560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:53137)
0924 10:02:19.783020 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.783176 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260924 10:02:19.885322 21466 raft_consensus.cc:482] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:19.885591 21466 raft_consensus.cc:504] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.887173 21466 leader_election.cc:288] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073), 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257)
I20260924 10:02:19.892660 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.887658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:45701 (local address 127.13.162.2:44257)
0924 10:02:19.887850 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.887856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.888036 (+ 180us) server_negotiation.cc:407] Connection header received
0924 10:02:19.888189 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.888194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.888262 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.888356 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.888746 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.889460 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.891059 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.891775 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.891931 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.892000 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.892286 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.892415 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:02:19.892663 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.887460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:44779 (local address 127.13.162.1:45073)
0924 10:02:19.887670 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.887676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.887698 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:19.887852 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.887858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.887931 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.888029 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:19.888506 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.889268 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.891005 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.891741 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.891860 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.891929 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.892304 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.892418 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:02:19.892683 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.887424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44257 (local address 127.13.162.3:45701)
0924 10:02:19.887927 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.887949 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.888052 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.888374 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.888377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.888395 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.888620 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.888630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.889589 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.889593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.890918 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.890929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.891975 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.891980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.892137 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.892173 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.892242 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.892380 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":172}
I20260924 10:02:19.892686 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.887379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:45073 (local address 127.13.162.3:44779)
0924 10:02:19.887558 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.887579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:19.887705 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:19.888050 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:19.888054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:19.888081 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:19.888369 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.888382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.889403 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.889408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.890858 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:19.890870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.891916 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.891926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:19.892124 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.892184 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 10:02:19.892265 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:02:19.892400 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":37}
I20260924 10:02:19.893445 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13237b3d2a964dbcb6e25bd99f0d59fe" candidate_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82add254d9d8479480e7018a651d2291" is_pre_election: true
I20260924 10:02:19.893709 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13237b3d2a964dbcb6e25bd99f0d59fe" candidate_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa49ed9187a42568c7ff491e12fb40b" is_pre_election: true
I20260924 10:02:19.893802 21173 raft_consensus.cc:2454] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86c2f01f67b41d09e1f94f36c9872cd in term 0.
I20260924 10:02:19.893946 21022 raft_consensus.cc:2454] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86c2f01f67b41d09e1f94f36c9872cd in term 0.
I20260924 10:02:19.894373 21247 leader_election.cc:302] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82add254d9d8479480e7018a651d2291, f86c2f01f67b41d09e1f94f36c9872cd; no voters:
I20260924 10:02:19.894755 21466 raft_consensus.cc:2781] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:19.894876 21466 raft_consensus.cc:482] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:19.894970 21466 raft_consensus.cc:3037] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:19.896981 21466 raft_consensus.cc:504] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.897500 21466 leader_election.cc:288] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [CANDIDATE]: Term 1 election: Requested vote from peers 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073), 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257)
I20260924 10:02:19.897929 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13237b3d2a964dbcb6e25bd99f0d59fe" candidate_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82add254d9d8479480e7018a651d2291"
I20260924 10:02:19.897938 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13237b3d2a964dbcb6e25bd99f0d59fe" candidate_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa49ed9187a42568c7ff491e12fb40b"
I20260924 10:02:19.898074 21173 raft_consensus.cc:3037] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:19.898100 21022 raft_consensus.cc:3037] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:19.900012 21173 raft_consensus.cc:2454] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86c2f01f67b41d09e1f94f36c9872cd in term 1.
I20260924 10:02:19.900005 21022 raft_consensus.cc:2454] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86c2f01f67b41d09e1f94f36c9872cd in term 1.
I20260924 10:02:19.900374 21247 leader_election.cc:302] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82add254d9d8479480e7018a651d2291, f86c2f01f67b41d09e1f94f36c9872cd; no voters:
I20260924 10:02:19.900636 21466 raft_consensus.cc:2781] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:19.901031 21466 raft_consensus.cc:686] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 1 LEADER]: Becoming Leader. State: Replica: f86c2f01f67b41d09e1f94f36c9872cd, State: Running, Role: LEADER
I20260924 10:02:19.901402 21466 consensus_queue.cc:237] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } }
I20260924 10:02:19.902032 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.901638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:56903)
0924 10:02:19.901787 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.901873 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:02:19.904680 20702 catalog_manager.cc:5495] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3). New cstate: current_term: 1 leader_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86c2f01f67b41d09e1f94f36c9872cd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 33439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260924 10:02:19.942830 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.937723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.13.162.3:33439)
0924 10:02:19.937896 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.937902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.937945 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:02:19.938196 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.938202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.938279 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.938462 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:19.938809 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.939536 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.940796 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.941219 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.941325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:19.941737 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:19.942276 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:19.942450 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:19.942495 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:02:19.942573 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:02:19.946678 21022 raft_consensus.cc:1261] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Refusing update from remote peer f86c2f01f67b41d09e1f94f36c9872cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:19.947007 21173 raft_consensus.cc:1261] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Refusing update from remote peer f86c2f01f67b41d09e1f94f36c9872cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:19.947314 21469 consensus_queue.cc:1035] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa49ed9187a42568c7ff491e12fb40b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:19.947763 21466 consensus_queue.cc:1035] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82add254d9d8479480e7018a651d2291" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:19.950230 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.949673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:40955)
0924 10:02:19.949906 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.950014 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 10:02:19.950243 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.949784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:45949)
0924 10:02:19.949995 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.950071 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 10:02:19.954619 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.954178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.1:41683)
0924 10:02:19.954390 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.954471 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 10:02:19.956466 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.956008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.2:37639)
0924 10:02:19.956222 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.956308 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 10:02:19.960134 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.959600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.13.162.3:38229)
0924 10:02:19.959915 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:19.960000 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260924 10:02:19.969868 20700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:19.970144 20700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:19.975167 20700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b7af120325d4809b3aff9cd14e1026c: Sending DeleteTablet for 3 replicas of tablet 13237b3d2a964dbcb6e25bd99f0d59fe
I20260924 10:02:19.976207 21002 tablet_service.cc:1514] Processing DeleteTablet for tablet 13237b3d2a964dbcb6e25bd99f0d59fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:19 UTC) from {username='oryx', principal='oryx/127.13.162.61@KRBTEST.COM'} at 127.0.0.1:43246
I20260924 10:02:19.976354 21153 tablet_service.cc:1514] Processing DeleteTablet for tablet 13237b3d2a964dbcb6e25bd99f0d59fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:19 UTC) from {username='oryx', principal='oryx/127.13.162.61@KRBTEST.COM'} at 127.0.0.1:60188
I20260924 10:02:19.976383 21294 tablet_service.cc:1514] Processing DeleteTablet for tablet 13237b3d2a964dbcb6e25bd99f0d59fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:19 UTC) from {username='oryx', principal='oryx/127.13.162.61@KRBTEST.COM'} at 127.0.0.1:44268
I20260924 10:02:19.977643 21482 tablet_replica.cc:330] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: stopping tablet replica
I20260924 10:02:19.977864 21482 raft_consensus.cc:2229] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:19.977913 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20868
I20260924 10:02:19.978201 21482 raft_consensus.cc:2258] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:19.983156 21482 ts_tablet_manager.cc:1905] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:19.988633 21483 tablet_replica.cc:330] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: stopping tablet replica
I20260924 10:02:19.988947 21483 raft_consensus.cc:2229] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:19.989319 21483 raft_consensus.cc:2258] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:19.989929 21482 ts_tablet_manager.cc:1918] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:19.990105 21482 log.cc:1196] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/13237b3d2a964dbcb6e25bd99f0d59fe
I20260924 10:02:19.990116 21484 tablet_replica.cc:330] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: stopping tablet replica
I20260924 10:02:19.990306 21484 raft_consensus.cc:2229] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:19.990469 21483 ts_tablet_manager.cc:1905] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:19.990603 21482 ts_tablet_manager.cc:1939] T 13237b3d2a964dbcb6e25bd99f0d59fe P 9fa49ed9187a42568c7ff491e12fb40b: Deleting consensus metadata
I20260924 10:02:19.990900 21484 raft_consensus.cc:2258] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:19.991773 20689 catalog_manager.cc:4841] TS 9fa49ed9187a42568c7ff491e12fb40b (127.13.162.1:45073): tablet 13237b3d2a964dbcb6e25bd99f0d59fe (table test-table [id=4129f8ba975b48a4972a23a77eb757d8]) successfully deleted
I20260924 10:02:19.991991 21484 ts_tablet_manager.cc:1905] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:19.995888 21483 ts_tablet_manager.cc:1918] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:19.996140 21483 log.cc:1196] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/13237b3d2a964dbcb6e25bd99f0d59fe
I20260924 10:02:19.996618 21483 ts_tablet_manager.cc:1939] T 13237b3d2a964dbcb6e25bd99f0d59fe P 82add254d9d8479480e7018a651d2291: Deleting consensus metadata
I20260924 10:02:19.997897 20686 catalog_manager.cc:4841] TS 82add254d9d8479480e7018a651d2291 (127.13.162.2:44257): tablet 13237b3d2a964dbcb6e25bd99f0d59fe (table test-table [id=4129f8ba975b48a4972a23a77eb757d8]) successfully deleted
I20260924 10:02:19.998756 21484 ts_tablet_manager.cc:1918] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:19.998900 21484 log.cc:1196] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/13237b3d2a964dbcb6e25bd99f0d59fe
I20260924 10:02:19.999310 21484 ts_tablet_manager.cc:1939] T 13237b3d2a964dbcb6e25bd99f0d59fe P f86c2f01f67b41d09e1f94f36c9872cd: Deleting consensus metadata
I20260924 10:02:20.000326 20689 catalog_manager.cc:4841] TS f86c2f01f67b41d09e1f94f36c9872cd (127.13.162.3:33439): tablet 13237b3d2a964dbcb6e25bd99f0d59fe (table test-table [id=4129f8ba975b48a4972a23a77eb757d8]) successfully deleted
I20260924 10:02:20.009333 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:19.979229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.13.162.1:45073)
0924 10:02:19.979399 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:19.979405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:19.993785 (+ 14380us) server_negotiation.cc:407] Connection header received
0924 10:02:19.994002 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:19.994007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:19.994088 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:19.994180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:19.996067 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.997136 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.999067 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:19.999681 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:19.999817 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:20.002642 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:20.002662 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:20.002685 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:20.002728 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:20.006424 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:20.007173 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:20.007183 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:20.007187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:20.007348 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:20.007669 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:20.007675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:20.007678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:20.008155 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:20.008469 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:20.008704 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:02:20.008953 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:02:20.010037 21062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:20.102061 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20868
I20260924 10:02:20.116403 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 21082
I20260924 10:02:20.130767 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:20.117176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.13.162.2:44257)
0924 10:02:20.117333 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:20.117337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:20.117391 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:02:20.117570 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:20.117574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:20.117635 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:20.117715 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:20.119480 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:20.120212 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:20.121968 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:20.122387 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:20.122494 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:20.125108 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:20.125125 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:20.125142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:20.125189 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:20.128067 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:20.128745 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:20.128753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:20.128755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:20.128873 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:20.129281 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:20.129284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:20.129286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:20.129622 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:20.129882 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:20.130175 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:02:20.130435 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:02:20.131417 21213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:20.217296 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:20.216818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37754)
0924 10:02:20.217019 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:20.217126 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:02:20.218557 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:20.218151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44465 (local address 127.0.0.1:37764)
0924 10:02:20.218304 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:20.218420 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:02:20.256101 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 21082
I20260924 10:02:20.274339 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 21231
I20260924 10:02:20.290221 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:20.275092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.13.162.3:33439)
0924 10:02:20.275278 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:20.275285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:20.275488 (+ 203us) server_negotiation.cc:407] Connection header received
0924 10:02:20.275657 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:20.275661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:20.275722 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:20.275814 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:20.277736 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:20.278711 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:20.280463 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:20.280898 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:20.280984 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:20.284357 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:20.284372 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:20.284391 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:20.284424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:20.287471 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:20.288296 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:20.288304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:20.288306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:20.288446 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:20.288834 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:20.288837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:20.288839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:20.289183 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:20.289412 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:20.289759 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 10:02:20.289934 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:02:20.290912 21354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:20.416445 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 21231
I20260924 10:02:20.432720 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20566
I20260924 10:02:20.433772 20732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:20.554507 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20566
I20260924 10:02:20.571199 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 20665
I20260924 10:02:20.572252 20920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:20.682997 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 20665
2026-09-24T10:02:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9139 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:20 dist-test-slave-jvsj krb5kdc[21493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:20 dist-test-slave-jvsj krb5kdc[21493](info): set up 2 sockets
Sep 24 10:02:20 dist-test-slave-jvsj krb5kdc[21493](info): commencing operation
krb5kdc: starting...
W20260924 10:02:21.694796 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:21 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab.
Entry for principal oryx/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab.
I20260924 10:02:21.889267 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44179
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab
--principal=oryx/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:22.195755 21506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:22.196031 21506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:22.196156 21506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:22.203244 21506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:22.203342 21506 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:02:22.203411 21506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:22.203495 21506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:22.203565 21506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:22.203635 21506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:22.213266 21506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44179
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.62.keytab
--principal=oryx/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:22.214531 21506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:22.215796 21506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:22.226747 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:22.226711 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:22.226785 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:22.227731 21506 server_base.cc:1034] running on GCE node
I20260924 10:02:22.228402 21506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:22.229712 21506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:22.230988 21506 hybrid_clock.cc:648] HybridClock initialized: now 1790244142230914 us; error 32 us; skew 500 ppm
I20260924 10:02:22.231316 21506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:22 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244142, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:22.238205 21506 init.cc:377] Logged in from keytab as oryx/127.13.162.62@KRBTEST.COM (short username oryx)
I20260924 10:02:22.239666 21506 webserver.cc:460] Webserver started at http://127.13.162.62:36757/ using document root <none> and password file <none>
I20260924 10:02:22.240327 21506 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:22.240412 21506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:22.240691 21506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:22.243557 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "162dff7be90a42e8b2d9ce74f1042d71"
format_stamp: "Formatted at 2026-09-24 10:02:22 on dist-test-slave-jvsj"
server_key: "ddfdf46c49f4c7012c003c62ea127293"
server_key_iv: "e3deb91660eb20d966425e5c3c3b72b2"
server_key_version: "encryptionkey@0"
I20260924 10:02:22.244354 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "162dff7be90a42e8b2d9ce74f1042d71"
format_stamp: "Formatted at 2026-09-24 10:02:22 on dist-test-slave-jvsj"
server_key: "ddfdf46c49f4c7012c003c62ea127293"
server_key_iv: "e3deb91660eb20d966425e5c3c3b72b2"
server_key_version: "encryptionkey@0"
I20260924 10:02:22.249399 21506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:02:22.252758 21521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:22.254109 21506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:22.254257 21506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "162dff7be90a42e8b2d9ce74f1042d71"
format_stamp: "Formatted at 2026-09-24 10:02:22 on dist-test-slave-jvsj"
server_key: "ddfdf46c49f4c7012c003c62ea127293"
server_key_iv: "e3deb91660eb20d966425e5c3c3b72b2"
server_key_version: "encryptionkey@0"
I20260924 10:02:22.254483 21506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:22.277967 21506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:22.281934 21506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:22.282169 21506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:22.301862 21506 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:44179
I20260924 10:02:22.301872 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:44179 every 8 connection(s)
I20260924 10:02:22.303205 21506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:22.306854 21583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:22.307490 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 21506
I20260924 10:02:22.307703 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:22.308532 13960 external_mini_cluster.cc:1428] Setting key f7d7de4663deed2b062a1648c03858b9
I20260924 10:02:22.320250 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Bootstrap starting.
Sep 24 10:02:22 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:22.323760 21583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:22.325027 21583 log.cc:824] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:22.328087 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: No bootstrap required, opened a new log
I20260924 10:02:22.331120 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:22.310572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.13.162.62:44179)
0924 10:02:22.311316 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:22.311327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:22.311370 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:02:22.312991 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:22.313013 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:22.313406 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:22.313743 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:22.315351 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.316643 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.318426 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.319110 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.319235 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:22.323451 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:22.323462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:22.323480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:22.323527 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:22.326909 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:22.327515 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:22.327523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:22.327531 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:22.327738 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:22.328058 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:22.328063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:22.328066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:22.328738 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:22.329013 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:22.329333 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 10:02:22.329750 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":340,"threads_started":1}
I20260924 10:02:22.336155 21583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162dff7be90a42e8b2d9ce74f1042d71" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44179 } }
I20260924 10:02:22.336457 21583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:22.336532 21583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162dff7be90a42e8b2d9ce74f1042d71, State: Initialized, Role: FOLLOWER
I20260924 10:02:22.337016 21583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162dff7be90a42e8b2d9ce74f1042d71" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44179 } }
I20260924 10:02:22.337211 21583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:22.337298 21583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:22.337415 21583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:22.339424 21583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162dff7be90a42e8b2d9ce74f1042d71" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44179 } }
I20260924 10:02:22.339902 21583 leader_election.cc:302] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 162dff7be90a42e8b2d9ce74f1042d71; no voters:
I20260924 10:02:22.340440 21583 leader_election.cc:288] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:22.340570 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:22.341261 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [term 1 LEADER]: Becoming Leader. State: Replica: 162dff7be90a42e8b2d9ce74f1042d71, State: Running, Role: LEADER
I20260924 10:02:22.341802 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162dff7be90a42e8b2d9ce74f1042d71" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44179 } }
I20260924 10:02:22.342365 21583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:22.344228 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162dff7be90a42e8b2d9ce74f1042d71. Latest consensus state: current_term: 1 leader_uuid: "162dff7be90a42e8b2d9ce74f1042d71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162dff7be90a42e8b2d9ce74f1042d71" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44179 } } }
I20260924 10:02:22.344607 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:22.348414 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "162dff7be90a42e8b2d9ce74f1042d71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162dff7be90a42e8b2d9ce74f1042d71" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44179 } } }
I20260924 10:02:22.348718 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71 [sys.catalog]: This master's current role is: LEADER
W20260924 10:02:22.351715 21603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:02:22.351809 21603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:02:22.351922 21595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:22.353960 21595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:22.362159 21595 catalog_manager.cc:1347] Generated new cluster ID: 30cbebe3d14847cb8822862bb5fa1997
I20260924 10:02:22.362255 21595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:22.391407 21595 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:22.392693 21595 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:22.412191 21595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Generated new TSK 0
I20260924 10:02:22.413041 21595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab.
Entry for principal oryx/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab.
I20260924 10:02:22.476281 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44179
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:22.788880 21609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:22.789155 21609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:22.789287 21609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:22.796346 21609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:22.796466 21609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:22.796595 21609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:02:22.796710 21609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:22.806243 21609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:22.807497 21609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:22.808779 21609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:22.820823 21614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:22.820785 21617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:22.820854 21615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:22.821548 21609 server_base.cc:1034] running on GCE node
I20260924 10:02:22.822098 21609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:22.823026 21609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:22.824223 21609 hybrid_clock.cc:648] HybridClock initialized: now 1790244142824150 us; error 50 us; skew 500 ppm
I20260924 10:02:22.824465 21609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:22 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244142, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:22.827387 21609 init.cc:377] Logged in from keytab as oryx/127.13.162.1@KRBTEST.COM (short username oryx)
I20260924 10:02:22.828701 21609 webserver.cc:460] Webserver started at http://127.13.162.1:45901/ using document root <none> and password file <none>
I20260924 10:02:22.829308 21609 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:22.829387 21609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:22.829638 21609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:22.832413 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "b9546c0961e644fc829a06a27dae4fd4"
format_stamp: "Formatted at 2026-09-24 10:02:22 on dist-test-slave-jvsj"
server_key: "778000d0d37d93cf477f1e74152d79da"
server_key_iv: "5bd363fd266e4a4103bdf95fa3c91a69"
server_key_version: "encryptionkey@0"
I20260924 10:02:22.833153 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "b9546c0961e644fc829a06a27dae4fd4"
format_stamp: "Formatted at 2026-09-24 10:02:22 on dist-test-slave-jvsj"
server_key: "778000d0d37d93cf477f1e74152d79da"
server_key_iv: "5bd363fd266e4a4103bdf95fa3c91a69"
server_key_version: "encryptionkey@0"
I20260924 10:02:22.837884 21609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:02:22.841089 21624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:22.842449 21609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:22.842590 21609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "b9546c0961e644fc829a06a27dae4fd4"
format_stamp: "Formatted at 2026-09-24 10:02:22 on dist-test-slave-jvsj"
server_key: "778000d0d37d93cf477f1e74152d79da"
server_key_iv: "5bd363fd266e4a4103bdf95fa3c91a69"
server_key_version: "encryptionkey@0"
I20260924 10:02:22.842787 21609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:22.859498 21609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:22.863507 21609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:22.863747 21609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:22.866338 21609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:22.866432 21609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:22.866570 21609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:22.866658 21609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:02:22 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244142, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:22.888602 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:22.866900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51449 (local address 127.13.162.62:44179)
0924 10:02:22.870685 (+ 3785us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:22.870691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:22.870712 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:02:22.871569 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:22.871575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:22.871656 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:22.871752 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:22.874018 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.875168 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.876729 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.877328 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.877461 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:22.882132 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:22.882139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:22.882143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:22.882183 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:22.885155 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:22.886103 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:22.886108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:22.886112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:22.886253 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:22.886779 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:22.886786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:22.886790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:22.887176 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:22.887319 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:22.887955 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 10:02:22.888194 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3656,"thread_start_us":3551,"threads_started":1}
I20260924 10:02:22.889369 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:22.867484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.13.162.1:51449)
0924 10:02:22.868415 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:22.868470 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:22.871205 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:22.872181 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:22.872194 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:22.872943 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:22.873766 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:22.873831 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.875308 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.875317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:22.876600 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:22.876611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.877584 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.877648 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:22.877935 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:22.878704 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:22.878730 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:22.881869 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:22.885307 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:22.885319 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:22.885335 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:22.885950 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:22.886369 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:22.886374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:22.886377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:22.886644 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:22.887316 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:22.887322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:22.887637 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 10:02:22.887861 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 10:02:22.888230 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":354,"threads_started":1}
I20260924 10:02:22.907778 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:22.894996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:32975 (local address 127.13.162.62:44179)
0924 10:02:22.895187 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:22.895193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:22.895711 (+ 518us) server_negotiation.cc:407] Connection header received
0924 10:02:22.895829 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:22.895834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:22.895916 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:22.896016 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:22.896796 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.897929 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.899230 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.899884 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.900003 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:22.902001 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:22.902006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:22.902009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:22.902037 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:22.904968 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:22.905584 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:22.905589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:22.905592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:22.905745 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:22.906176 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:22.906181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:22.906184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:22.906690 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:22.906826 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:22.907220 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 10:02:22.907428 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:02:22.907847 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:22.895030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.13.162.1:32975)
0924 10:02:22.895624 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:22.895641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:22.895744 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:22.896019 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:22.896022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:22.896411 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:22.896687 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:22.896695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.898048 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.898051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:22.899094 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:22.899106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:22.900057 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:22.900061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:22.900193 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:22.900825 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:22.900841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:22.901840 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:22.905082 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:22.905087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:22.905091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:22.905471 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:22.905888 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:22.905892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:22.905896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:22.906083 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:22.906812 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:22.906817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:22.906992 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 10:02:22.907110 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:02:22.907425 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":306,"threads_started":1}
I20260924 10:02:22.916337 21609 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:37691
I20260924 10:02:22.916364 21744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:37691 every 8 connection(s)
I20260924 10:02:22.917478 21609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:22.919310 21745 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44179
I20260924 10:02:22.919510 21745 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:22.920104 21745 heartbeater.cc:511] Master 127.13.162.62:44179 requested a full tablet report, sending...
I20260924 10:02:22.921893 21538 ts_manager.cc:194] Registered new tserver with Master: b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691)
I20260924 10:02:22.923300 21538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:51449
I20260924 10:02:22.926348 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 21609
I20260924 10:02:22.926522 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:22.927139 13960 external_mini_cluster.cc:1428] Setting key 5daa2afaf957b9e56d55345e3f0753f0
WARNING: no policy specified for oryx/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab.
Entry for principal oryx/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab.
I20260924 10:02:22.967428 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44179
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:23.270567 21749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:23.270830 21749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:23.270920 21749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:23.277797 21749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:23.277895 21749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:23.277987 21749 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:02:23.278061 21749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:23.287353 21749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:23.288661 21749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:23.290311 21749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:23.303460 21757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:23.303709 21755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:23.303920 21749 server_base.cc:1034] running on GCE node
W20260924 10:02:23.303468 21754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:23.304755 21749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:23.305783 21749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:23.307019 21749 hybrid_clock.cc:648] HybridClock initialized: now 1790244143306958 us; error 32 us; skew 500 ppm
I20260924 10:02:23.307339 21749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244143, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:23.310892 21749 init.cc:377] Logged in from keytab as oryx/127.13.162.2@KRBTEST.COM (short username oryx)
I20260924 10:02:23.312387 21749 webserver.cc:460] Webserver started at http://127.13.162.2:34819/ using document root <none> and password file <none>
I20260924 10:02:23.313184 21749 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:23.313305 21749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:23.313612 21749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:23.317361 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "42cd00c6a5a744b4b541c5d738a2d7c6"
format_stamp: "Formatted at 2026-09-24 10:02:23 on dist-test-slave-jvsj"
server_key: "f806d9633eabb8c51e1879ce00429413"
server_key_iv: "13294fb4d78ad623904f2928fae86410"
server_key_version: "encryptionkey@0"
I20260924 10:02:23.318303 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "42cd00c6a5a744b4b541c5d738a2d7c6"
format_stamp: "Formatted at 2026-09-24 10:02:23 on dist-test-slave-jvsj"
server_key: "f806d9633eabb8c51e1879ce00429413"
server_key_iv: "13294fb4d78ad623904f2928fae86410"
server_key_version: "encryptionkey@0"
I20260924 10:02:23.324213 21749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 10:02:23.327596 21764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:23.328910 21749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:02:23.329062 21749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "42cd00c6a5a744b4b541c5d738a2d7c6"
format_stamp: "Formatted at 2026-09-24 10:02:23 on dist-test-slave-jvsj"
server_key: "f806d9633eabb8c51e1879ce00429413"
server_key_iv: "13294fb4d78ad623904f2928fae86410"
server_key_version: "encryptionkey@0"
I20260924 10:02:23.329253 21749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:23.345328 21749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:23.349628 21749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:23.349864 21749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:23.352440 21749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:23.352541 21749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:23.352697 21749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:23.352770 21749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244143, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:23.374055 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.352904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:56235 (local address 127.13.162.62:44179)
0924 10:02:23.353074 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:23.353081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:23.354497 (+ 1416us) server_negotiation.cc:407] Connection header received
0924 10:02:23.356614 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:23.356621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:23.356697 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:23.356780 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:23.358385 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.359364 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.361781 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.362403 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.362522 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.367084 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:23.367092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:23.367097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:23.367140 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:23.370483 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.371381 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.371387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.371391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.371513 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.372008 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.372013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.372017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.372370 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:23.372517 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:23.373406 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 10:02:23.373663 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:02:23.375186 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.353511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.13.162.2:56235)
0924 10:02:23.354353 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:23.354394 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:23.356225 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:23.357035 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:23.357043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:23.357541 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:23.358196 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.358214 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.359536 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.359540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.361592 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.361605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.362853 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.362860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.363117 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.363812 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:23.363844 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:23.366837 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:23.370657 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.370665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.370688 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.371253 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.371632 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.371636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.371640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.371905 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.372495 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:23.372502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:23.372998 (+ 496us) client_negotiation.cc:769] Sending connection context
0924 10:02:23.373299 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 10:02:23.373754 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":388,"threads_started":1}
I20260924 10:02:23.395282 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.381685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:48089 (local address 127.13.162.62:44179)
0924 10:02:23.381888 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:23.381894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:23.381921 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:02:23.382134 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:23.382140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:23.382221 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:23.382342 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:23.383252 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.384379 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.386059 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.386687 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.386816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.389233 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:23.389240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:23.389245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:23.389294 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:23.391994 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.392964 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.392969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.392972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.393131 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.393564 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.393568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.393571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.394013 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:23.394166 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:23.394706 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 10:02:23.394945 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:02:23.395675 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.381297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.13.162.2:48089)
0924 10:02:23.381837 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:23.381860 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:23.381997 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:23.382413 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:23.382418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:23.382864 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:23.383131 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.383139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.384508 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.384514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.385925 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.385937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.386904 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.386911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.387106 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.387931 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:23.387958 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:23.389025 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:23.392151 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.392157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.392161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.392803 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.393253 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.393261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.393264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.393441 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.394141 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:23.394146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:23.394409 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 10:02:23.394608 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 10:02:23.394999 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260924 10:02:23.399500 21749 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:43077
I20260924 10:02:23.399521 21883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:43077 every 8 connection(s)
I20260924 10:02:23.400646 21749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:23.402489 21884 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44179
I20260924 10:02:23.402680 21884 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:23.403268 21884 heartbeater.cc:511] Master 127.13.162.62:44179 requested a full tablet report, sending...
I20260924 10:02:23.404467 21538 ts_manager.cc:194] Registered new tserver with Master: 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077)
I20260924 10:02:23.405205 21538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:56235
I20260924 10:02:23.407310 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 21749
I20260924 10:02:23.407450 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:23.407886 13960 external_mini_cluster.cc:1428] Setting key d22cf349148192ef343253e42a68be39
WARNING: no policy specified for oryx/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab.
Entry for principal oryx/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab.
I20260924 10:02:23.448648 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44179
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:23.757331 21888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:23.757614 21888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:23.757731 21888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:23.764977 21888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:23.765079 21888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:23.765197 21888 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:02:23.765298 21888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:23.774770 21888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.162.62:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:23.775993 21888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:23.777348 21888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:23.794353 21894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:23.794513 21893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:23.795298 21896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:23.796276 21888 server_base.cc:1034] running on GCE node
I20260924 10:02:23.796983 21888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:23.797961 21888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:23.799254 21888 hybrid_clock.cc:648] HybridClock initialized: now 1790244143799154 us; error 56 us; skew 500 ppm
I20260924 10:02:23.799606 21888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244143, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:23.803591 21888 init.cc:377] Logged in from keytab as oryx/127.13.162.3@KRBTEST.COM (short username oryx)
I20260924 10:02:23.805099 21888 webserver.cc:460] Webserver started at http://127.13.162.3:44361/ using document root <none> and password file <none>
I20260924 10:02:23.805788 21888 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:23.805876 21888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:23.806176 21888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:23.809736 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "918efb90e64340fc9b848757541e4f19"
format_stamp: "Formatted at 2026-09-24 10:02:23 on dist-test-slave-jvsj"
server_key: "f9fa8a22ea1d74f36b478c6c616b1c95"
server_key_iv: "4445bf67276bcdf2873258cbaa33ce5a"
server_key_version: "encryptionkey@0"
I20260924 10:02:23.810737 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "918efb90e64340fc9b848757541e4f19"
format_stamp: "Formatted at 2026-09-24 10:02:23 on dist-test-slave-jvsj"
server_key: "f9fa8a22ea1d74f36b478c6c616b1c95"
server_key_iv: "4445bf67276bcdf2873258cbaa33ce5a"
server_key_version: "encryptionkey@0"
I20260924 10:02:23.817005 21888 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260924 10:02:23.821334 21903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:23.823158 21888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:23.823305 21888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "918efb90e64340fc9b848757541e4f19"
format_stamp: "Formatted at 2026-09-24 10:02:23 on dist-test-slave-jvsj"
server_key: "f9fa8a22ea1d74f36b478c6c616b1c95"
server_key_iv: "4445bf67276bcdf2873258cbaa33ce5a"
server_key_version: "encryptionkey@0"
I20260924 10:02:23.823504 21888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:23.859949 21888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:23.863845 21888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:23.864091 21888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:23.866677 21888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:23.866773 21888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:23.866915 21888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:23.866976 21888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244143, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:23.890209 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.868152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.13.162.3:49187)
0924 10:02:23.868928 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:23.868963 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:23.870760 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:23.871503 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:23.871515 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:23.872244 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:23.873033 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.873074 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.874364 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.874370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.876358 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.876369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.877330 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.877339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.877692 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.878452 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:23.878493 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:23.881812 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:23.885187 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.885196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.885217 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.885952 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.886389 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.886394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.886399 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.886682 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.887414 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:23.887420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:23.887880 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 10:02:23.888156 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 10:02:23.888677 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":351,"threads_started":1}
I20260924 10:02:23.891289 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.867642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:49187 (local address 127.13.162.62:44179)
0924 10:02:23.867805 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:23.867809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:23.869072 (+ 1263us) server_negotiation.cc:407] Connection header received
0924 10:02:23.871041 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:23.871046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:23.871130 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:23.871239 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:23.873230 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.874194 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.876481 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.876987 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.877106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.882025 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:23.882032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:23.882037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:23.882079 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:23.885044 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.886096 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.886102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.886105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.886261 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.886830 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.886835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.886838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.887221 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:23.887357 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:23.890711 (+ 3354us) server_negotiation.cc:299] Negotiation successful
0924 10:02:23.890929 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:02:23.908159 21888 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:35171
I20260924 10:02:23.908181 22022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:35171 every 8 connection(s)
I20260924 10:02:23.909621 21888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:23.909935 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.897060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.13.162.3:32935)
0924 10:02:23.897604 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:23.897623 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:23.897738 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:23.898011 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:23.898016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:23.898478 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:23.898909 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.898919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.900122 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.900127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.901532 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:23.901547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.902515 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.902521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:23.902635 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.903184 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:23.903199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:23.903860 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:23.907388 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.907394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.907398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.908007 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.908443 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:23.908446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:23.908449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:23.908560 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:23.909068 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:23.909071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:23.909215 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 10:02:23.909317 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:02:23.909525 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":299,"threads_started":1}
I20260924 10:02:23.909951 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:23.896931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:32935 (local address 127.13.162.62:44179)
0924 10:02:23.897075 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:23.897081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:23.897697 (+ 616us) server_negotiation.cc:407] Connection header received
0924 10:02:23.897839 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:23.897844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:23.897908 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:23.897980 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:23.899052 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.899953 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.901674 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.902299 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.902439 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:23.904042 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:23.904048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:23.904054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:23.904091 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:23.907242 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.908170 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.908174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.908178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.908306 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:23.908658 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:23.908663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:23.908667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:23.908970 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:23.909117 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:23.909425 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:02:23.909641 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:02:23.912039 22023 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44179
I20260924 10:02:23.912240 22023 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:23.912997 22023 heartbeater.cc:511] Master 127.13.162.62:44179 requested a full tablet report, sending...
I20260924 10:02:23.914379 21538 ts_manager.cc:194] Registered new tserver with Master: 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:23.915100 21538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:49187
I20260924 10:02:23.917609 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 21888
I20260924 10:02:23.917742 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:23.918160 13960 external_mini_cluster.cc:1428] Setting key d3d0a008c0375ed9416da6464b4136bf
I20260924 10:02:23.920791 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:02:23.925782 21745 heartbeater.cc:503] Master 127.13.162.62:44179 was elected leader, sending a full tablet report...
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244143, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:23 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:24.407400 21884 heartbeater.cc:503] Master 127.13.162.62:44179 was elected leader, sending a full tablet report...
I20260924 10:02:24.917260 22023 heartbeater.cc:503] Master 127.13.162.62:44179 was elected leader, sending a full tablet report...
W20260924 10:02:24.924036 22031 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.964s user 0.004s sys 0.008s
W20260924 10:02:24.925056 21633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:23.954952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.13.162.62:44179)
0924 10:02:23.955174 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:23.955180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:23.955404 (+ 224us) server_negotiation.cc:407] Connection header received
0924 10:02:23.955534 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:23.955538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:23.955606 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:23.955689 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:23.956889 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.957705 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.958964 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:23.959379 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:23.959465 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.924773 (+965308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35966: BlockingRecv error: recv got EOF from 127.0.0.1:35966 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 10:02:24 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244143, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:24.941002 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.928985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.13.162.62:44179)
0924 10:02:24.929153 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:24.929158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:24.929365 (+ 207us) server_negotiation.cc:407] Connection header received
0924 10:02:24.929504 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:24.929507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:24.929590 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:24.929661 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:24.930826 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.931526 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.932752 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.933129 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.933212 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.936630 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:24.936635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:24.936638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:24.936667 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:24.938935 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:24.939461 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:24.939465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:24.939468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:24.939579 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:24.939885 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:24.939890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:24.939892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:24.940159 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:24.940378 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:24.940534 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:02:24.940690 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:02:24.944903 21538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:24.951655 21538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:24.977053 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.962841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.13.162.2:43077)
0924 10:02:24.969479 (+ 6638us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:24.969487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:24.969513 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:02:24.969608 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:24.969616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:24.969883 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:24.970759 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:24.970971 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.972204 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.974779 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.975988 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.976128 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.976328 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:24.976592 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 10:02:24.976760 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6466,"thread_start_us":6300,"threads_started":1}
I20260924 10:02:24.977108 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.962621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:43077 (local address 127.0.0.1:50924)
0924 10:02:24.968688 (+ 6067us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:24.968729 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:24.968907 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:24.970120 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:24.970128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:24.970158 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:24.970559 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:24.970571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.972362 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.972367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:24.974639 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:24.974650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.976167 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.976173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:24.976350 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.976406 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 10:02:24.976532 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 10:02:24.976708 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5804,"mutex_wait_us":102,"thread_start_us":6829,"threads_started":1}
I20260924 10:02:24.977939 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.963811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.13.162.3:35171)
0924 10:02:24.964404 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:24.964414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:24.969846 (+ 5432us) server_negotiation.cc:407] Connection header received
0924 10:02:24.970153 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:24.970162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:24.970416 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:24.970581 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:24.971457 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.972746 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.975276 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.976646 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.976775 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.976963 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:24.977471 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 10:02:24.977664 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":267,"threads_started":1}
I20260924 10:02:24.977968 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.963230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:35171 (local address 127.0.0.1:50848)
0924 10:02:24.969740 (+ 6510us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:24.969756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:24.969958 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:24.970802 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:24.970806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:24.970835 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:24.971275 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:24.971285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.972875 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.972879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:24.975126 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:24.975139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.977121 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.977129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:24.977280 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.977312 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:02:24.977412 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 10:02:24.977534 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6375,"mutex_wait_us":3,"thread_start_us":5030,"threads_started":1}
I20260924 10:02:24.978175 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.970021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.13.162.1:37691)
0924 10:02:24.970627 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:24.970635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:24.971290 (+ 655us) server_negotiation.cc:407] Connection header received
0924 10:02:24.971451 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:24.971458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:24.971644 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:24.971818 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:24.972297 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.973450 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.975710 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.976900 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.977034 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.977218 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:24.977748 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 10:02:24.977898 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":319,"threads_started":1}
I20260924 10:02:24.978824 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:24.970646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37691 (local address 127.0.0.1:40012)
0924 10:02:24.971174 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:24.971189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:24.971308 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:24.971824 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:24.971829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:24.971852 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:24.972137 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:24.972146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.973610 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.973615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:24.975585 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:24.975595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:24.977384 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:24.977390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:24.977544 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:24.977579 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:02:24.978350 (+ 771us) client_negotiation.cc:240] Negotiation successful
0924 10:02:24.978471 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":19,"thread_start_us":260,"threads_started":1}
I20260924 10:02:24.981539 21675 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9f7c9d8a6942398b55f66b5e5843aa (DEFAULT_TABLE table=test-table [id=c8132bb608194f938d6e43eb801f7d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:24.981700 21953 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9f7c9d8a6942398b55f66b5e5843aa (DEFAULT_TABLE table=test-table [id=c8132bb608194f938d6e43eb801f7d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:24.981699 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9f7c9d8a6942398b55f66b5e5843aa (DEFAULT_TABLE table=test-table [id=c8132bb608194f938d6e43eb801f7d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:24.982738 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9f7c9d8a6942398b55f66b5e5843aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:24.982995 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9f7c9d8a6942398b55f66b5e5843aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:24.982995 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9f7c9d8a6942398b55f66b5e5843aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:24.993417 22048 tablet_bootstrap.cc:492] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Bootstrap starting.
I20260924 10:02:24.996517 22048 tablet_bootstrap.cc:654] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:24.997462 22048 log.cc:824] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:24.998649 22049 tablet_bootstrap.cc:492] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Bootstrap starting.
I20260924 10:02:25.000416 22048 tablet_bootstrap.cc:492] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: No bootstrap required, opened a new log
I20260924 10:02:25.000671 22048 ts_tablet_manager.cc:1397] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:02:25.002019 22049 tablet_bootstrap.cc:654] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.003000 22050 tablet_bootstrap.cc:492] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Bootstrap starting.
I20260924 10:02:25.003163 22049 log.cc:824] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:25.005659 22050 tablet_bootstrap.cc:654] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.006863 22050 log.cc:824] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:25.008386 22049 tablet_bootstrap.cc:492] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: No bootstrap required, opened a new log
I20260924 10:02:25.008651 22049 ts_tablet_manager.cc:1397] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 10:02:25.009142 22050 tablet_bootstrap.cc:492] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: No bootstrap required, opened a new log
I20260924 10:02:25.009359 22050 ts_tablet_manager.cc:1397] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:02:25.010905 22048 raft_consensus.cc:348] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.011313 22048 raft_consensus.cc:374] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.011404 22048 raft_consensus.cc:729] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9546c0961e644fc829a06a27dae4fd4, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.012102 22048 consensus_queue.cc:260] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.014906 22048 ts_tablet_manager.cc:1428] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 10:02:25.017421 22049 raft_consensus.cc:348] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.017819 22049 raft_consensus.cc:374] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.017921 22049 raft_consensus.cc:729] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.017845 22050 raft_consensus.cc:348] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.018356 22050 raft_consensus.cc:374] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.018476 22050 raft_consensus.cc:729] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.018565 22049 consensus_queue.cc:260] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.019034 22050 consensus_queue.cc:260] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.020237 22049 ts_tablet_manager.cc:1428] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 10:02:25.021674 22050 ts_tablet_manager.cc:1428] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 10:02:25.043941 22055 raft_consensus.cc:482] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:25.044239 22055 raft_consensus.cc:504] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.045989 22055 leader_election.cc:288] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077), b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691)
I20260924 10:02:25.052112 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.046348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:40543 (local address 127.13.162.2:43077)
0924 10:02:25.046574 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.046581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.046986 (+ 405us) server_negotiation.cc:407] Connection header received
0924 10:02:25.047177 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.047182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.047262 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.047356 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:25.048073 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.048804 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.050515 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.051286 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.051434 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.051507 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.051775 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.051894 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:02:25.052449 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.046260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:43077 (local address 127.13.162.3:40543)
0924 10:02:25.046883 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.046901 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.047037 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.047526 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.047532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.047567 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:25.047943 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.047953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.048935 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.048939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.050377 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.050390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.051476 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.051482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.051634 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.051658 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.051735 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.051860 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":3,"thread_start_us":370,"threads_started":1}
I20260924 10:02:25.052448 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.047222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:34383 (local address 127.13.162.1:37691)
0924 10:02:25.047387 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.047393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.047888 (+ 495us) server_negotiation.cc:407] Connection header received
0924 10:02:25.048050 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.048055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.048110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.048180 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:25.048655 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.049322 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.050917 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.051662 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.051795 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.051860 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.052117 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.052234 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:02:25.052518 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.047068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37691 (local address 127.13.162.3:34383)
0924 10:02:25.047792 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.047805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.047924 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.048199 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.048204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.048227 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:25.048519 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.048527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.049444 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.049450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.050774 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.050786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.051855 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.051861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.051984 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.052010 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.052084 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.052200 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":463,"threads_started":1}
I20260924 10:02:25.053393 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9f7c9d8a6942398b55f66b5e5843aa" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4" is_pre_election: true
I20260924 10:02:25.053402 21838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9f7c9d8a6942398b55f66b5e5843aa" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" is_pre_election: true
I20260924 10:02:25.053814 21838 raft_consensus.cc:2454] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 0.
I20260924 10:02:25.053817 21695 raft_consensus.cc:2454] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 0.
I20260924 10:02:25.054355 21906 leader_election.cc:302] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918efb90e64340fc9b848757541e4f19, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:25.054698 22055 raft_consensus.cc:2781] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:25.054814 22055 raft_consensus.cc:482] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:25.054903 22055 raft_consensus.cc:3037] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.056931 22055 raft_consensus.cc:504] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.057441 22055 leader_election.cc:288] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 election: Requested vote from peers 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077), b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691)
I20260924 10:02:25.057826 21838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9f7c9d8a6942398b55f66b5e5843aa" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6"
I20260924 10:02:25.057911 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9f7c9d8a6942398b55f66b5e5843aa" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4"
I20260924 10:02:25.058002 21838 raft_consensus.cc:3037] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.058091 21695 raft_consensus.cc:3037] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.060513 21838 raft_consensus.cc:2454] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 1.
I20260924 10:02:25.060643 21695 raft_consensus.cc:2454] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 1.
I20260924 10:02:25.060833 21906 leader_election.cc:302] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cd00c6a5a744b4b541c5d738a2d7c6, 918efb90e64340fc9b848757541e4f19; no voters:
I20260924 10:02:25.061105 22055 raft_consensus.cc:2781] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:25.061476 22055 raft_consensus.cc:686] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 1 LEADER]: Becoming Leader. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Running, Role: LEADER
I20260924 10:02:25.061913 22055 consensus_queue.cc:237] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.066097 21538 catalog_manager.cc:5495] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 918efb90e64340fc9b848757541e4f19 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } health_report { overall_health: HEALTHY } } }
I20260924 10:02:25.095490 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.090433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.13.162.3:35171)
0924 10:02:25.090643 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.090648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.090668 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:02:25.090790 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.090797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.090883 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.091017 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:25.091364 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.092102 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.093449 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.093837 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.093953 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.094392 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.094933 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.095072 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.095155 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.095235 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:02:25.103214 21838 raft_consensus.cc:1261] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Refusing update from remote peer 918efb90e64340fc9b848757541e4f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.103214 21695 raft_consensus.cc:1261] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Refusing update from remote peer 918efb90e64340fc9b848757541e4f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.104156 22059 consensus_queue.cc:1035] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.104512 22055 consensus_queue.cc:1035] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.116636 22064 mvcc.cc:205] Tried to move back new op lower bound from 7332840018333106176 to 7332840018176999424. Current Snapshot: MvccSnapshot[applied={T|T < 7332840018333106176}]
I20260924 10:02:25.128760 21538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:25.128968 21538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:25.132179 21538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Sending DeleteTablet for 3 replicas of tablet 7e9f7c9d8a6942398b55f66b5e5843aa
I20260924 10:02:25.133282 21953 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9f7c9d8a6942398b55f66b5e5843aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50848
I20260924 10:02:25.133356 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9f7c9d8a6942398b55f66b5e5843aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50924
I20260924 10:02:25.133910 21675 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9f7c9d8a6942398b55f66b5e5843aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260924 10:02:25.134160 22072 tablet_replica.cc:330] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: stopping tablet replica
I20260924 10:02:25.134625 22072 raft_consensus.cc:2229] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:25.135313 22072 raft_consensus.cc:2258] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.135563 21538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:25.136219 21538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:25.137637 22072 ts_tablet_manager.cc:1905] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.143563 21674 tablet_service.cc:1467] Processing CreateTablet for tablet f125c6e41bd44b169025d510cab4016c (DEFAULT_TABLE table=test-table [id=065e13caa5d14342bc3d76ed4f054fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.143755 21813 tablet_service.cc:1467] Processing CreateTablet for tablet f125c6e41bd44b169025d510cab4016c (DEFAULT_TABLE table=test-table [id=065e13caa5d14342bc3d76ed4f054fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.144109 21674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f125c6e41bd44b169025d510cab4016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.144282 21813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f125c6e41bd44b169025d510cab4016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.145067 22072 ts_tablet_manager.cc:1918] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:25.145292 22072 log.cc:1196] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/7e9f7c9d8a6942398b55f66b5e5843aa
I20260924 10:02:25.145843 22072 ts_tablet_manager.cc:1939] T 7e9f7c9d8a6942398b55f66b5e5843aa P 918efb90e64340fc9b848757541e4f19: Deleting consensus metadata
I20260924 10:02:25.146884 21953 tablet_service.cc:1467] Processing CreateTablet for tablet f125c6e41bd44b169025d510cab4016c (DEFAULT_TABLE table=test-table [id=065e13caa5d14342bc3d76ed4f054fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.147303 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f125c6e41bd44b169025d510cab4016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.147934 21525 catalog_manager.cc:4841] TS 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171): tablet 7e9f7c9d8a6942398b55f66b5e5843aa (table test-table [id=c8132bb608194f938d6e43eb801f7d4c]) successfully deleted
I20260924 10:02:25.150672 22050 tablet_bootstrap.cc:492] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Bootstrap starting.
I20260924 10:02:25.152208 22050 tablet_bootstrap.cc:654] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.154940 22073 tablet_replica.cc:330] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: stopping tablet replica
I20260924 10:02:25.155370 22073 raft_consensus.cc:2229] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:25.155846 22073 raft_consensus.cc:2258] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.157670 22073 ts_tablet_manager.cc:1905] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.163117 22049 tablet_bootstrap.cc:492] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Bootstrap starting.
I20260924 10:02:25.164526 22048 tablet_bootstrap.cc:492] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Bootstrap starting.
I20260924 10:02:25.164618 22073 ts_tablet_manager.cc:1918] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:25.164757 22073 log.cc:1196] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/7e9f7c9d8a6942398b55f66b5e5843aa
I20260924 10:02:25.164783 22049 tablet_bootstrap.cc:654] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.165261 22073 ts_tablet_manager.cc:1939] T 7e9f7c9d8a6942398b55f66b5e5843aa P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting consensus metadata
I20260924 10:02:25.166133 22048 tablet_bootstrap.cc:654] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.167003 21524 catalog_manager.cc:4841] TS 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077): tablet 7e9f7c9d8a6942398b55f66b5e5843aa (table test-table [id=c8132bb608194f938d6e43eb801f7d4c]) successfully deleted
I20260924 10:02:25.175868 22048 tablet_bootstrap.cc:492] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: No bootstrap required, opened a new log
I20260924 10:02:25.176033 22048 ts_tablet_manager.cc:1397] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 10:02:25.176579 22049 tablet_bootstrap.cc:492] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: No bootstrap required, opened a new log
I20260924 10:02:25.176774 22049 ts_tablet_manager.cc:1397] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260924 10:02:25.176837 22048 raft_consensus.cc:348] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.177062 22048 raft_consensus.cc:374] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.177145 22048 raft_consensus.cc:729] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9546c0961e644fc829a06a27dae4fd4, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.177417 22048 consensus_queue.cc:260] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.177701 22049 raft_consensus.cc:348] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.177911 22049 raft_consensus.cc:374] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.178016 22049 raft_consensus.cc:729] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.178092 22048 ts_tablet_manager.cc:1428] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 10:02:25.178357 22049 consensus_queue.cc:260] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.179888 22049 ts_tablet_manager.cc:1428] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:25.180827 22074 tablet_replica.cc:330] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: stopping tablet replica
I20260924 10:02:25.181094 22050 tablet_bootstrap.cc:492] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: No bootstrap required, opened a new log
I20260924 10:02:25.181234 22050 ts_tablet_manager.cc:1397] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.010s
I20260924 10:02:25.181263 22074 raft_consensus.cc:2229] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:25.181710 22074 raft_consensus.cc:2258] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.182159 22050 raft_consensus.cc:348] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.182405 22050 raft_consensus.cc:374] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.182492 22050 raft_consensus.cc:729] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.182796 22050 consensus_queue.cc:260] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.183621 22074 ts_tablet_manager.cc:1905] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.183777 22050 ts_tablet_manager.cc:1428] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:02:25.188426 22074 ts_tablet_manager.cc:1918] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:25.188567 22074 log.cc:1196] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/7e9f7c9d8a6942398b55f66b5e5843aa
I20260924 10:02:25.188942 22074 ts_tablet_manager.cc:1939] T 7e9f7c9d8a6942398b55f66b5e5843aa P b9546c0961e644fc829a06a27dae4fd4: Deleting consensus metadata
I20260924 10:02:25.190395 21524 catalog_manager.cc:4841] TS b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691): tablet 7e9f7c9d8a6942398b55f66b5e5843aa (table test-table [id=c8132bb608194f938d6e43eb801f7d4c]) successfully deleted
I20260924 10:02:25.241291 22056 raft_consensus.cc:482] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:25.241528 22056 raft_consensus.cc:504] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.243441 22056 leader_election.cc:288] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691), 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:25.250897 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.243952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:39873 (local address 127.13.162.1:37691)
0924 10:02:25.244132 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.244138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.244813 (+ 675us) server_negotiation.cc:407] Connection header received
0924 10:02:25.245012 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.245018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.245090 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.245171 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:25.245922 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.247017 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.249007 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.249910 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.250071 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.250141 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.250498 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.250632 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:02:25.250975 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.243807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37691 (local address 127.13.162.2:39873)
0924 10:02:25.244725 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.244742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.244863 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.245346 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.245350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.245369 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:25.245766 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.245776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.247169 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.247172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.248858 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.248869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.250125 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.250132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.250300 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.250359 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.250445 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.250576 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":245,"thread_start_us":510,"threads_started":1}
I20260924 10:02:25.251689 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f125c6e41bd44b169025d510cab4016c" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4" is_pre_election: true
I20260924 10:02:25.251942 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.243952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:60931 (local address 127.13.162.3:35171)
0924 10:02:25.244132 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.244138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.244749 (+ 611us) server_negotiation.cc:407] Connection header received
0924 10:02:25.244915 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.244921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.245019 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.245114 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:25.245791 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.246956 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.249575 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.250684 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.250821 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.250885 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.251574 (+ 689us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.251698 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:02:25.251950 21695 raft_consensus.cc:2454] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 0.
I20260924 10:02:25.252597 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.243807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:35171 (local address 127.13.162.2:60931)
0924 10:02:25.244639 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.244658 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.244799 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.245139 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.245143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.245173 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:25.245570 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.245581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.247114 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.247120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.249410 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.249424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.251051 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.251214 (+ 163us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.251393 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.251434 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.252112 (+ 678us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.252242 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":47,"thread_start_us":498,"threads_started":1}
I20260924 10:02:25.252620 21767 leader_election.cc:302] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cd00c6a5a744b4b541c5d738a2d7c6, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:25.253047 22056 raft_consensus.cc:2781] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:25.253160 22056 raft_consensus.cc:482] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:25.253247 22056 raft_consensus.cc:3037] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.253362 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f125c6e41bd44b169025d510cab4016c" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918efb90e64340fc9b848757541e4f19" is_pre_election: true
I20260924 10:02:25.253813 21973 raft_consensus.cc:2454] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 0.
I20260924 10:02:25.255481 22056 raft_consensus.cc:504] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.256011 22056 leader_election.cc:288] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 election: Requested vote from peers b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691), 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:25.256446 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f125c6e41bd44b169025d510cab4016c" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918efb90e64340fc9b848757541e4f19"
I20260924 10:02:25.256469 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f125c6e41bd44b169025d510cab4016c" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4"
I20260924 10:02:25.256615 21973 raft_consensus.cc:3037] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.256690 21695 raft_consensus.cc:3037] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.259272 21973 raft_consensus.cc:2454] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 1.
I20260924 10:02:25.259558 21695 raft_consensus.cc:2454] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 1.
I20260924 10:02:25.259869 21768 leader_election.cc:302] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cd00c6a5a744b4b541c5d738a2d7c6, 918efb90e64340fc9b848757541e4f19; no voters:
I20260924 10:02:25.260134 22056 raft_consensus.cc:2781] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:25.260577 22056 raft_consensus.cc:686] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 LEADER]: Becoming Leader. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Running, Role: LEADER
I20260924 10:02:25.261093 22056 consensus_queue.cc:237] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.265267 21538 catalog_manager.cc:5495] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2). New cstate: current_term: 1 leader_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:02:25 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244142, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.62@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:25.324851 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.308759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.13.162.62:44179)
0924 10:02:25.308974 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.308980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.309621 (+ 641us) server_negotiation.cc:407] Connection header received
0924 10:02:25.309778 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.309784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.309859 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.309972 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:25.311459 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.312622 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.314404 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.315060 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.315168 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.319268 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:25.319274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:25.319279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:25.319312 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:25.322102 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:25.322952 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:25.322956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:25.322958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:25.323082 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:25.323440 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:25.323443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:25.323445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:25.323701 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:25.323816 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.324262 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.324544 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":91}
I20260924 10:02:25.324911 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.308861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44179 (local address 127.0.0.1:35990)
0924 10:02:25.309504 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.309523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.309671 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.310000 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.310005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.310922 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:25.311308 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.311323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.312749 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.312754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.314254 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.314267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.315243 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.315249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.315421 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.316151 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:25.316171 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:25.319062 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:25.322218 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:25.322229 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:25.322236 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:25.322848 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:25.323178 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:25.323182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:25.323186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:25.323348 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:25.323820 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:25.323828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:25.324016 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.324182 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.324443 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":351,"threads_started":1}
I20260924 10:02:25.330024 21538 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:35990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:02:25.331260 21538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:25.337687 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 6008f3e44d78486494c5330a26f761bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e93e6ba683d0456d8bbad1c2061e5023]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:02:25.337808 21953 tablet_service.cc:1467] Processing CreateTablet for tablet 6008f3e44d78486494c5330a26f761bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e93e6ba683d0456d8bbad1c2061e5023]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:02:25.337904 21675 tablet_service.cc:1467] Processing CreateTablet for tablet 6008f3e44d78486494c5330a26f761bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e93e6ba683d0456d8bbad1c2061e5023]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:02:25.338261 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6008f3e44d78486494c5330a26f761bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.338384 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6008f3e44d78486494c5330a26f761bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.338378 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6008f3e44d78486494c5330a26f761bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.345288 22050 tablet_bootstrap.cc:492] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: Bootstrap starting.
I20260924 10:02:25.345297 22048 tablet_bootstrap.cc:492] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: Bootstrap starting.
I20260924 10:02:25.346917 22050 tablet_bootstrap.cc:654] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.346935 22048 tablet_bootstrap.cc:654] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.347152 22049 tablet_bootstrap.cc:492] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: Bootstrap starting.
I20260924 10:02:25.348821 22049 tablet_bootstrap.cc:654] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.351107 22050 tablet_bootstrap.cc:492] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: No bootstrap required, opened a new log
I20260924 10:02:25.351253 22050 ts_tablet_manager.cc:1397] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:02:25.351783 22050 raft_consensus.cc:348] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.351965 22050 raft_consensus.cc:374] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.352038 22050 raft_consensus.cc:729] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.352265 22050 consensus_queue.cc:260] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.352664 22056 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.352910 22056 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.353368 22050 ts_tablet_manager.cc:1428] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 10:02:25.363720 22048 tablet_bootstrap.cc:492] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: No bootstrap required, opened a new log
I20260924 10:02:25.363905 22048 ts_tablet_manager.cc:1397] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260924 10:02:25.364085 22049 tablet_bootstrap.cc:492] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: No bootstrap required, opened a new log
I20260924 10:02:25.364259 22049 ts_tablet_manager.cc:1397] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.014s
I20260924 10:02:25.364648 22048 raft_consensus.cc:348] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.364886 22048 raft_consensus.cc:374] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.364965 22048 raft_consensus.cc:729] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9546c0961e644fc829a06a27dae4fd4, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.364993 22049 raft_consensus.cc:348] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.365212 22049 raft_consensus.cc:374] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.365295 22049 raft_consensus.cc:729] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.365234 22048 consensus_queue.cc:260] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.365521 22049 consensus_queue.cc:260] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.365665 22054 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.366005 22054 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.365947 22067 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.366274 22067 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.366605 22048 ts_tablet_manager.cc:1428] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:25.366763 22049 ts_tablet_manager.cc:1428] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260924 10:02:25.404218 21885 tablet.cc:2369] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:02:25.412357 22024 tablet.cc:2369] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:02:25.421342 21746 tablet.cc:2369] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:02:25.538635 22067 raft_consensus.cc:482] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:25.538882 22067 raft_consensus.cc:504] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.539425 22067 leader_election.cc:288] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077), b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691)
I20260924 10:02:25.540012 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6008f3e44d78486494c5330a26f761bc" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4" is_pre_election: true
I20260924 10:02:25.540023 21838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6008f3e44d78486494c5330a26f761bc" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" is_pre_election: true
I20260924 10:02:25.540251 21695 raft_consensus.cc:2454] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 0.
I20260924 10:02:25.540288 21838 raft_consensus.cc:2454] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 0.
I20260924 10:02:25.540642 21906 leader_election.cc:302] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cd00c6a5a744b4b541c5d738a2d7c6, 918efb90e64340fc9b848757541e4f19; no voters:
I20260924 10:02:25.540931 22067 raft_consensus.cc:2781] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:25.541051 22067 raft_consensus.cc:482] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:25.541152 22067 raft_consensus.cc:3037] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.543572 22067 raft_consensus.cc:504] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.544045 22067 leader_election.cc:288] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 election: Requested vote from peers 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077), b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691)
I20260924 10:02:25.544564 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6008f3e44d78486494c5330a26f761bc" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4"
I20260924 10:02:25.544577 21838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6008f3e44d78486494c5330a26f761bc" candidate_uuid: "918efb90e64340fc9b848757541e4f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6"
I20260924 10:02:25.544752 21695 raft_consensus.cc:3037] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.544767 21838 raft_consensus.cc:3037] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.546844 21695 raft_consensus.cc:2454] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 1.
I20260924 10:02:25.547225 21906 leader_election.cc:302] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918efb90e64340fc9b848757541e4f19, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:25.547487 22067 raft_consensus.cc:2781] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:25.547619 22067 raft_consensus.cc:686] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [term 1 LEADER]: Becoming Leader. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Running, Role: LEADER
I20260924 10:02:25.547824 21838 raft_consensus.cc:2454] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918efb90e64340fc9b848757541e4f19 in term 1.
I20260924 10:02:25.547865 22067 consensus_queue.cc:237] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } }
I20260924 10:02:25.549568 22055 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: TxnStatusTablet state changed. Reason: New leader 918efb90e64340fc9b848757541e4f19. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.549831 22055 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:02:25.550693 22092 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:02:25.550904 22092 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:02:25.550971 21536 catalog_manager.cc:5495] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 918efb90e64340fc9b848757541e4f19 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } health_report { overall_health: UNKNOWN } } }
I20260924 10:02:25.594489 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.588729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:35171 (local address 127.0.0.1:50860)
0924 10:02:25.589117 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.589136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.589279 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.589711 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.589715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.589724 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:02:25.589909 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.589916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.591381 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.591385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.592477 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.592489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.593410 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.593414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.593554 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.593606 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.594028 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.594045 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.594108 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.594219 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:02:25.594563 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.588797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.13.162.3:35171)
0924 10:02:25.588970 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.588976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.589221 (+ 245us) server_negotiation.cc:407] Connection header received
0924 10:02:25.589444 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.589451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.589547 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.589687 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:25.590057 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.591256 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.592603 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.593284 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.593408 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.593759 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.593890 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.593987 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.594158 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.594236 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:02:25.597818 21695 raft_consensus.cc:1261] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Refusing update from remote peer 918efb90e64340fc9b848757541e4f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.597864 21838 raft_consensus.cc:1261] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Refusing update from remote peer 918efb90e64340fc9b848757541e4f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.598397 22067 consensus_queue.cc:1035] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.598672 22055 consensus_queue.cc:1035] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.601677 22056 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: TxnStatusTablet state changed. Reason: New leader 918efb90e64340fc9b848757541e4f19. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.601936 22056 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.603674 22067 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.603932 22067 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:02:25.607547 22062 mvcc.cc:205] Tried to move back new op lower bound from 7332840020362682368 to 7332840020166287360. Current Snapshot: MvccSnapshot[applied={T|T < 7332840020362682368 or (T in {7332840020362682368})}]
I20260924 10:02:25.608759 22063 mvcc.cc:205] Tried to move back new op lower bound from 7332840020362682368 to 7332840020166287360. Current Snapshot: MvccSnapshot[applied={T|T < 7332840020362682368}]
I20260924 10:02:25.609143 22056 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.609403 22056 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 42cd00c6a5a744b4b541c5d738a2d7c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.613953 22054 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: TxnStatusTablet state changed. Reason: New leader 918efb90e64340fc9b848757541e4f19. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.614248 22054 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.615573 22055 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.615839 22055 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P 918efb90e64340fc9b848757541e4f19: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:02:25.616825 22054 tablet_replica.cc:439] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "918efb90e64340fc9b848757541e4f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } } peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } }
I20260924 10:02:25.617035 22054 tablet_replica.cc:442] T 6008f3e44d78486494c5330a26f761bc P b9546c0961e644fc829a06a27dae4fd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:02:25.624816 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.615703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.13.162.2:43077)
0924 10:02:25.618767 (+ 3064us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.618773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.618795 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:25.618886 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.618891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.618990 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.619157 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:25.619704 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.620868 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.622396 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.623031 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.623157 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.623511 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.624220 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.624356 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.624401 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.624479 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2931,"thread_start_us":2796,"threads_started":1}
I20260924 10:02:25.640398 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.634553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:35171 (local address 127.13.162.2:51503)
0924 10:02:25.635100 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.635114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.635229 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.635542 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.635545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.635553 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:02:25.635809 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.635816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.637217 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.637221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.638401 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.638412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.639377 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.639381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.639495 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.639557 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.639950 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.639956 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.640005 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.640101 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":304,"threads_started":1}
I20260924 10:02:25.640437 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.634503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:51503 (local address 127.13.162.3:35171)
0924 10:02:25.634687 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.634691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.635201 (+ 510us) server_negotiation.cc:407] Connection header received
0924 10:02:25.635351 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.635357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.635445 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.635538 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:25.635930 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.637089 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.638527 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.639226 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.639354 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.639704 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.639864 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.639945 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.640042 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.640122 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:02:25.651103 21973 raft_consensus.cc:1261] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Refusing update from remote peer 42cd00c6a5a744b4b541c5d738a2d7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.651110 21695 raft_consensus.cc:1261] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Refusing update from remote peer 42cd00c6a5a744b4b541c5d738a2d7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.651713 22056 consensus_queue.cc:1035] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.651997 22080 consensus_queue.cc:1035] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.688120 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.681092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:43077 (local address 127.13.162.3:46485)
0924 10:02:25.681692 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:25.681711 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:25.681815 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:25.682127 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:25.682132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:25.682145 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:02:25.682514 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.682526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.684610 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.684614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.685916 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:25.685931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.686938 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.686945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:25.687131 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.687211 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.687627 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.687638 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 10:02:25.687698 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:02:25.687798 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":281,"threads_started":1}
I20260924 10:02:25.688113 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.681077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46485 (local address 127.13.162.2:43077)
0924 10:02:25.681246 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.681252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.681782 (+ 530us) server_negotiation.cc:407] Connection header received
0924 10:02:25.681905 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.681911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.681989 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.682080 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:25.682724 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.684464 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.686040 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.686688 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.686800 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.687394 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.687514 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.687591 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.687741 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.687818 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:02:25.721967 21538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:25.722241 21538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:25.725679 21538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Sending DeleteTablet for 3 replicas of tablet f125c6e41bd44b169025d510cab4016c
I20260924 10:02:25.726574 21675 tablet_service.cc:1514] Processing DeleteTablet for tablet f125c6e41bd44b169025d510cab4016c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260924 10:02:25.726974 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet f125c6e41bd44b169025d510cab4016c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50924
I20260924 10:02:25.728215 21953 tablet_service.cc:1514] Processing DeleteTablet for tablet f125c6e41bd44b169025d510cab4016c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50848
I20260924 10:02:25.736860 22110 tablet_replica.cc:330] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: stopping tablet replica
I20260924 10:02:25.737244 22110 raft_consensus.cc:2229] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:25.737650 22110 raft_consensus.cc:2258] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.738853 22110 ts_tablet_manager.cc:1905] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.746287 22110 ts_tablet_manager.cc:1918] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:02:25.746624 22110 log.cc:1196] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/f125c6e41bd44b169025d510cab4016c
I20260924 10:02:25.747159 22110 ts_tablet_manager.cc:1939] T f125c6e41bd44b169025d510cab4016c P b9546c0961e644fc829a06a27dae4fd4: Deleting consensus metadata
I20260924 10:02:25.748471 21524 catalog_manager.cc:4841] TS b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691): tablet f125c6e41bd44b169025d510cab4016c (table test-table [id=065e13caa5d14342bc3d76ed4f054fc6]) successfully deleted
I20260924 10:02:25.759200 22111 tablet_replica.cc:330] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: stopping tablet replica
I20260924 10:02:25.759589 22111 raft_consensus.cc:2229] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:25.758770 22112 tablet_replica.cc:330] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: stopping tablet replica
I20260924 10:02:25.759975 22112 raft_consensus.cc:2229] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:25.760700 22111 raft_consensus.cc:2258] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.761062 22112 raft_consensus.cc:2258] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.762382 22112 ts_tablet_manager.cc:1905] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.762905 22111 ts_tablet_manager.cc:1905] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.770364 22111 ts_tablet_manager.cc:1918] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:02:25.770658 22111 log.cc:1196] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/f125c6e41bd44b169025d510cab4016c
I20260924 10:02:25.771137 22111 ts_tablet_manager.cc:1939] T f125c6e41bd44b169025d510cab4016c P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting consensus metadata
I20260924 10:02:25.772451 21524 catalog_manager.cc:4841] TS 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077): tablet f125c6e41bd44b169025d510cab4016c (table test-table [id=065e13caa5d14342bc3d76ed4f054fc6]) successfully deleted
I20260924 10:02:25.780323 22112 ts_tablet_manager.cc:1918] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:02:25.780473 22112 log.cc:1196] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/f125c6e41bd44b169025d510cab4016c
I20260924 10:02:25.780934 22112 ts_tablet_manager.cc:1939] T f125c6e41bd44b169025d510cab4016c P 918efb90e64340fc9b848757541e4f19: Deleting consensus metadata
I20260924 10:02:25.782027 21525 catalog_manager.cc:4841] TS 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171): tablet f125c6e41bd44b169025d510cab4016c (table test-table [id=065e13caa5d14342bc3d76ed4f054fc6]) successfully deleted
Sep 24 10:02:25 dist-test-slave-jvsj krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:02:25 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:25 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:02:25.808120 21633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:25.794999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.13.162.62:44179)
0924 10:02:25.795176 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.795183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.795710 (+ 527us) server_negotiation.cc:407] Connection header received
0924 10:02:25.795878 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.795884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.795961 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.796057 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:25.797647 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.798780 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.800579 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.801232 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.801352 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.807841 (+ 6489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35994: BlockingRecv error: recv got EOF from 127.0.0.1:35994 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:02:25 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM
I20260924 10:02:25.834961 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.815931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.13.162.62:44179)
0924 10:02:25.816124 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.816129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.816704 (+ 575us) server_negotiation.cc:407] Connection header received
0924 10:02:25.816780 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.816786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.816869 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.816971 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:25.818598 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.819702 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.821779 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.822465 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.822588 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.827523 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:25.827532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:25.827537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:25.827582 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:25.831161 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:25.831990 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:25.831996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:25.832000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:25.832153 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:25.832607 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:25.832612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:25.832616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:25.832990 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:25.833226 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.834368 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.834592 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:02:25.838959 21538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:25.839668 21538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:25.847398 21675 tablet_service.cc:1467] Processing CreateTablet for tablet d01a6ff02a464e06a87eed7f352abb0a (DEFAULT_TABLE table=test-table [id=3b556b39fa414212bb26c090702fe753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.847879 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01a6ff02a464e06a87eed7f352abb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.849289 21953 tablet_service.cc:1467] Processing CreateTablet for tablet d01a6ff02a464e06a87eed7f352abb0a (DEFAULT_TABLE table=test-table [id=3b556b39fa414212bb26c090702fe753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.849599 21814 tablet_service.cc:1467] Processing CreateTablet for tablet d01a6ff02a464e06a87eed7f352abb0a (DEFAULT_TABLE table=test-table [id=3b556b39fa414212bb26c090702fe753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.849761 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01a6ff02a464e06a87eed7f352abb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.849972 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01a6ff02a464e06a87eed7f352abb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.854393 22048 tablet_bootstrap.cc:492] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Bootstrap starting.
I20260924 10:02:25.855995 22048 tablet_bootstrap.cc:654] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.856238 22049 tablet_bootstrap.cc:492] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Bootstrap starting.
I20260924 10:02:25.857986 22049 tablet_bootstrap.cc:654] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.861266 22048 tablet_bootstrap.cc:492] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: No bootstrap required, opened a new log
I20260924 10:02:25.861447 22048 ts_tablet_manager.cc:1397] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 10:02:25.862365 22048 raft_consensus.cc:348] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.864367 22048 raft_consensus.cc:374] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.864483 22048 raft_consensus.cc:729] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9546c0961e644fc829a06a27dae4fd4, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.864773 22048 consensus_queue.cc:260] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.865494 22048 ts_tablet_manager.cc:1428] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 10:02:25.872336 22049 tablet_bootstrap.cc:492] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: No bootstrap required, opened a new log
I20260924 10:02:25.872494 22049 ts_tablet_manager.cc:1397] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.009s
I20260924 10:02:25.873179 22049 raft_consensus.cc:348] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.873391 22049 raft_consensus.cc:374] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.873483 22049 raft_consensus.cc:729] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.873739 22049 consensus_queue.cc:260] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.874388 22049 ts_tablet_manager.cc:1428] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:02:25.877373 22126 tablet_bootstrap.cc:492] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Bootstrap starting.
I20260924 10:02:25.879647 22126 tablet_bootstrap.cc:654] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:25.881732 22126 tablet_bootstrap.cc:492] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: No bootstrap required, opened a new log
I20260924 10:02:25.881879 22126 ts_tablet_manager.cc:1397] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 10:02:25.882637 22126 raft_consensus.cc:348] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.882843 22126 raft_consensus.cc:374] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:25.882936 22126 raft_consensus.cc:729] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Initialized, Role: FOLLOWER
I20260924 10:02:25.883172 22126 consensus_queue.cc:260] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.883772 22126 ts_tablet_manager.cc:1428] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:02:25.896468 22080 raft_consensus.cc:482] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:25.896682 22080 raft_consensus.cc:504] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.897339 22080 leader_election.cc:288] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691), 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:25.897871 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01a6ff02a464e06a87eed7f352abb0a" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4" is_pre_election: true
I20260924 10:02:25.898093 21695 raft_consensus.cc:2454] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 0.
I20260924 10:02:25.898615 21767 leader_election.cc:302] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cd00c6a5a744b4b541c5d738a2d7c6, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:25.898729 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01a6ff02a464e06a87eed7f352abb0a" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918efb90e64340fc9b848757541e4f19" is_pre_election: true
I20260924 10:02:25.898973 21973 raft_consensus.cc:2454] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 0.
I20260924 10:02:25.899039 22080 raft_consensus.cc:2781] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:25.899120 22080 raft_consensus.cc:482] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:25.899223 22080 raft_consensus.cc:3037] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.901497 22080 raft_consensus.cc:504] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.902494 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01a6ff02a464e06a87eed7f352abb0a" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9546c0961e644fc829a06a27dae4fd4"
I20260924 10:02:25.902608 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01a6ff02a464e06a87eed7f352abb0a" candidate_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918efb90e64340fc9b848757541e4f19"
I20260924 10:02:25.902700 21695 raft_consensus.cc:3037] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.902786 21973 raft_consensus.cc:3037] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:25.904556 22080 leader_election.cc:288] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 election: Requested vote from peers b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691), 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:25.905421 21695 raft_consensus.cc:2454] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 1.
I20260924 10:02:25.905947 21767 leader_election.cc:302] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cd00c6a5a744b4b541c5d738a2d7c6, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:25.906117 21973 raft_consensus.cc:2454] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cd00c6a5a744b4b541c5d738a2d7c6 in term 1.
I20260924 10:02:25.907073 22080 raft_consensus.cc:2781] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:25.907294 22080 raft_consensus.cc:686] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 LEADER]: Becoming Leader. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Running, Role: LEADER
I20260924 10:02:25.907579 22080 consensus_queue.cc:237] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:25.909993 21536 catalog_manager.cc:5495] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2). New cstate: current_term: 1 leader_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } health_report { overall_health: UNKNOWN } } }
I20260924 10:02:25.942965 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:25.936311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.13.162.2:43077)
0924 10:02:25.936492 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:25.936498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:25.936516 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:25.936587 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:25.936593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:25.936688 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:25.936777 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:25.937308 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.938490 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.940454 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:25.941084 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:25.941199 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:25.941766 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:25.941894 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:25.941984 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:25.942567 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 10:02:25.942644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:02:25.946116 21973 raft_consensus.cc:1261] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Refusing update from remote peer 42cd00c6a5a744b4b541c5d738a2d7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.946978 22107 consensus_queue.cc:1035] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.948649 21695 raft_consensus.cc:1261] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Refusing update from remote peer 42cd00c6a5a744b4b541c5d738a2d7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:25.949318 22107 consensus_queue.cc:1035] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:25.978021 21538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:25.978381 21538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:25.981700 21538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Sending DeleteTablet for 3 replicas of tablet d01a6ff02a464e06a87eed7f352abb0a
I20260924 10:02:25.982437 21675 tablet_service.cc:1514] Processing DeleteTablet for tablet d01a6ff02a464e06a87eed7f352abb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260924 10:02:25.982704 22110 tablet_replica.cc:330] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: stopping tablet replica
I20260924 10:02:25.982923 22110 raft_consensus.cc:2229] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:25.983134 22110 raft_consensus.cc:2258] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.983592 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet d01a6ff02a464e06a87eed7f352abb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50924
I20260924 10:02:25.983866 22111 tablet_replica.cc:330] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: stopping tablet replica
I20260924 10:02:25.984061 22111 raft_consensus.cc:2229] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:25.984027 21953 tablet_service.cc:1514] Processing DeleteTablet for tablet d01a6ff02a464e06a87eed7f352abb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:25 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50848
I20260924 10:02:25.984282 22112 tablet_replica.cc:330] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: stopping tablet replica
I20260924 10:02:25.984368 22111 raft_consensus.cc:2258] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.984453 22112 raft_consensus.cc:2229] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:25.984669 22112 raft_consensus.cc:2258] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:25.985147 21536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:02:25.985513 22110 ts_tablet_manager.cc:1905] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:25.985647 22112 ts_tablet_manager.cc:1905] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:02:25.986032 21536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:25.991806 22112 ts_tablet_manager.cc:1918] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:25.991809 22110 ts_tablet_manager.cc:1918] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:25.991946 22112 log.cc:1196] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/d01a6ff02a464e06a87eed7f352abb0a
I20260924 10:02:25.991946 22110 log.cc:1196] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/d01a6ff02a464e06a87eed7f352abb0a
I20260924 10:02:25.992317 22112 ts_tablet_manager.cc:1939] T d01a6ff02a464e06a87eed7f352abb0a P 918efb90e64340fc9b848757541e4f19: Deleting consensus metadata
I20260924 10:02:25.992323 22110 ts_tablet_manager.cc:1939] T d01a6ff02a464e06a87eed7f352abb0a P b9546c0961e644fc829a06a27dae4fd4: Deleting consensus metadata
I20260924 10:02:25.993588 21953 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93d3c58f274af4aecc585061ee5a99 (DEFAULT_TABLE table=test-table [id=c4056cbb6cb04510b4fd4012e224c856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.993988 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93d3c58f274af4aecc585061ee5a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:25.998150 21525 catalog_manager.cc:4841] TS 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171): tablet d01a6ff02a464e06a87eed7f352abb0a (table test-table [id=3b556b39fa414212bb26c090702fe753]) successfully deleted
I20260924 10:02:25.998698 21524 catalog_manager.cc:4841] TS b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691): tablet d01a6ff02a464e06a87eed7f352abb0a (table test-table [id=3b556b39fa414212bb26c090702fe753]) successfully deleted
I20260924 10:02:25.999106 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93d3c58f274af4aecc585061ee5a99 (DEFAULT_TABLE table=test-table [id=c4056cbb6cb04510b4fd4012e224c856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:25.999554 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93d3c58f274af4aecc585061ee5a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:26.000412 22111 ts_tablet_manager.cc:1905] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:26.005057 21675 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93d3c58f274af4aecc585061ee5a99 (DEFAULT_TABLE table=test-table [id=c4056cbb6cb04510b4fd4012e224c856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:26.005499 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93d3c58f274af4aecc585061ee5a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:26.007438 22049 tablet_bootstrap.cc:492] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Bootstrap starting.
I20260924 10:02:26.009068 22049 tablet_bootstrap.cc:654] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:26.010233 22111 ts_tablet_manager.cc:1918] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:26.011556 22111 log.cc:1196] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/d01a6ff02a464e06a87eed7f352abb0a
I20260924 10:02:26.011842 22048 tablet_bootstrap.cc:492] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Bootstrap starting.
I20260924 10:02:26.012526 22126 tablet_bootstrap.cc:492] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Bootstrap starting.
I20260924 10:02:26.013295 22048 tablet_bootstrap.cc:654] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:26.015338 22126 tablet_bootstrap.cc:654] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:26.018985 22111 ts_tablet_manager.cc:1939] T d01a6ff02a464e06a87eed7f352abb0a P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting consensus metadata
I20260924 10:02:26.019666 22049 tablet_bootstrap.cc:492] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: No bootstrap required, opened a new log
I20260924 10:02:26.019908 22049 ts_tablet_manager.cc:1397] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260924 10:02:26.020110 21524 catalog_manager.cc:4841] TS 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077): tablet d01a6ff02a464e06a87eed7f352abb0a (table test-table [id=3b556b39fa414212bb26c090702fe753]) successfully deleted
I20260924 10:02:26.020825 22049 raft_consensus.cc:348] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.021167 22049 raft_consensus.cc:374] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:26.021281 22049 raft_consensus.cc:729] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918efb90e64340fc9b848757541e4f19, State: Initialized, Role: FOLLOWER
I20260924 10:02:26.021595 22049 consensus_queue.cc:260] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.022825 22126 tablet_bootstrap.cc:492] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: No bootstrap required, opened a new log
I20260924 10:02:26.022990 22126 ts_tablet_manager.cc:1397] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260924 10:02:26.023224 22049 ts_tablet_manager.cc:1428] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 10:02:26.023758 22126 raft_consensus.cc:348] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.024013 22048 tablet_bootstrap.cc:492] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: No bootstrap required, opened a new log
I20260924 10:02:26.024024 22126 raft_consensus.cc:374] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:26.024094 22126 raft_consensus.cc:729] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cd00c6a5a744b4b541c5d738a2d7c6, State: Initialized, Role: FOLLOWER
I20260924 10:02:26.024195 22048 ts_tablet_manager.cc:1397] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260924 10:02:26.024331 22126 consensus_queue.cc:260] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.024878 22126 ts_tablet_manager.cc:1428] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 10:02:26.026680 22048 raft_consensus.cc:348] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.027064 22048 raft_consensus.cc:374] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:26.027189 22048 raft_consensus.cc:729] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9546c0961e644fc829a06a27dae4fd4, State: Initialized, Role: FOLLOWER
I20260924 10:02:26.027469 22048 consensus_queue.cc:260] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.028012 22048 ts_tablet_manager.cc:1428] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:26.155861 22054 raft_consensus.cc:482] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:26.156076 22054 raft_consensus.cc:504] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.157544 22054 leader_election.cc:288] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077), 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:26.175658 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.157913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:39557 (local address 127.13.162.3:35171)
0924 10:02:26.167944 (+ 10031us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.167952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.168728 (+ 776us) server_negotiation.cc:407] Connection header received
0924 10:02:26.168847 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.168853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.168933 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.169051 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:26.169649 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.170666 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.173179 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.174395 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.174559 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.174636 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.175261 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.175389 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9637,"thread_start_us":9822,"threads_started":1}
I20260924 10:02:26.175631 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.158247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:34983 (local address 127.13.162.2:43077)
0924 10:02:26.158467 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.158473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.168533 (+ 10060us) server_negotiation.cc:407] Connection header received
0924 10:02:26.168823 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.168829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.168894 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.168939 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:26.169739 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.170601 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.173444 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.174461 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.174595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.174669 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.175229 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.175364 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 10:02:26.175985 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.157848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:35171 (local address 127.13.162.1:39557)
0924 10:02:26.168624 (+ 10776us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:26.168638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:26.168744 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:26.169083 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:26.169087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:26.169127 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:26.169479 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.169491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.170797 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.170802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.173010 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.173025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.174574 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.174581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.174753 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.174784 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:02:26.174877 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:02:26.175007 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10675,"mutex_wait_us":32,"spinlock_wait_cycles":1280,"thread_start_us":10266,"threads_started":1}
I20260924 10:02:26.176568 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.157803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:43077 (local address 127.13.162.1:34983)
0924 10:02:26.168399 (+ 10596us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:26.168422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:26.168656 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:26.169090 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:26.169094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:26.169135 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:26.169573 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.169583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.170720 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.170725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.173282 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.173294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.174866 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.174872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.175028 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.175056 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:02:26.176068 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0924 10:02:26.176198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10466,"mutex_wait_us":344,"spinlock_wait_cycles":13056,"thread_start_us":10547,"threads_started":1}
I20260924 10:02:26.176704 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93d3c58f274af4aecc585061ee5a99" candidate_uuid: "b9546c0961e644fc829a06a27dae4fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918efb90e64340fc9b848757541e4f19" is_pre_election: true
I20260924 10:02:26.176990 21973 raft_consensus.cc:2454] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9546c0961e644fc829a06a27dae4fd4 in term 0.
I20260924 10:02:26.177217 21838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93d3c58f274af4aecc585061ee5a99" candidate_uuid: "b9546c0961e644fc829a06a27dae4fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" is_pre_election: true
I20260924 10:02:26.177434 21838 raft_consensus.cc:2454] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9546c0961e644fc829a06a27dae4fd4 in term 0.
I20260924 10:02:26.177517 21628 leader_election.cc:302] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918efb90e64340fc9b848757541e4f19, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:26.177943 22054 raft_consensus.cc:2781] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:26.178062 22054 raft_consensus.cc:482] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:26.178167 22054 raft_consensus.cc:3037] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:26.180199 22054 raft_consensus.cc:504] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.180758 22054 leader_election.cc:288] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077), 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171)
I20260924 10:02:26.181259 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93d3c58f274af4aecc585061ee5a99" candidate_uuid: "b9546c0961e644fc829a06a27dae4fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918efb90e64340fc9b848757541e4f19"
I20260924 10:02:26.181329 21838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93d3c58f274af4aecc585061ee5a99" candidate_uuid: "b9546c0961e644fc829a06a27dae4fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6"
I20260924 10:02:26.181452 21973 raft_consensus.cc:3037] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:26.181506 21838 raft_consensus.cc:3037] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:26.184191 21973 raft_consensus.cc:2454] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9546c0961e644fc829a06a27dae4fd4 in term 1.
I20260924 10:02:26.184412 21838 raft_consensus.cc:2454] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9546c0961e644fc829a06a27dae4fd4 in term 1.
I20260924 10:02:26.184554 21628 leader_election.cc:302] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918efb90e64340fc9b848757541e4f19, b9546c0961e644fc829a06a27dae4fd4; no voters:
I20260924 10:02:26.184815 22054 raft_consensus.cc:2781] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:26.185247 22054 raft_consensus.cc:686] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 1 LEADER]: Becoming Leader. State: Replica: b9546c0961e644fc829a06a27dae4fd4, State: Running, Role: LEADER
I20260924 10:02:26.185766 22054 consensus_queue.cc:237] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } }
I20260924 10:02:26.189536 21538 catalog_manager.cc:5495] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1). New cstate: current_term: 1 leader_uuid: "b9546c0961e644fc829a06a27dae4fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9546c0961e644fc829a06a27dae4fd4" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 } health_report { overall_health: UNKNOWN } } }
I20260924 10:02:26.230932 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.224673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.13.162.1:37691)
0924 10:02:26.225177 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.225182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.225204 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:26.225295 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.225300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.225401 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.225569 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:26.225867 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.227057 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.228411 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.229041 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.229159 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.229634 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:26.230355 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:26.230496 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.230556 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.230628 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260924 10:02:26.246081 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.240800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:50733 (local address 127.13.162.3:35171)
0924 10:02:26.240973 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.240977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.241480 (+ 503us) server_negotiation.cc:407] Connection header received
0924 10:02:26.241628 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.241636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.241697 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.241770 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:26.242358 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.243110 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.244467 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.244854 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.244982 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.245383 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:26.245479 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:26.245554 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.245701 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.245773 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:02:26.246192 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.240794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:35171 (local address 127.13.162.1:50733)
0924 10:02:26.241404 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:26.241419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:26.241531 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:26.241890 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:26.241894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:26.241904 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:02:26.242203 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.242214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.243237 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.243241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.244352 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.244363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.245061 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.245066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.245192 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.245247 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:02:26.245579 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:02:26.245590 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 10:02:26.245670 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:02:26.245797 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":300,"threads_started":1}
I20260924 10:02:26.254204 21973 raft_consensus.cc:1261] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Refusing update from remote peer b9546c0961e644fc829a06a27dae4fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:26.254431 21838 raft_consensus.cc:1261] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Refusing update from remote peer b9546c0961e644fc829a06a27dae4fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:26.254884 22139 consensus_queue.cc:1035] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "918efb90e64340fc9b848757541e4f19" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:26.255301 22054 consensus_queue.cc:1035] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cd00c6a5a744b4b541c5d738a2d7c6" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:26.283975 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.274776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:54329 (local address 127.13.162.1:37691)
0924 10:02:26.274943 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.274949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.278802 (+ 3853us) server_negotiation.cc:407] Connection header received
0924 10:02:26.278917 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.278921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.278977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.279055 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:26.279439 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.280319 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.281919 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.282476 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.282592 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.283020 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:26.283158 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:26.283225 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.283627 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.283704 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:26.284072 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.274358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37691 (local address 127.13.162.3:54329)
0924 10:02:26.278711 (+ 4353us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:26.278726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:26.278829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:26.279092 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:26.279097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:26.279109 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:02:26.279319 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.279328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.280436 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.280441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.281792 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:26.281804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.282617 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.282622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:26.282826 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.282871 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:02:26.283499 (+ 628us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:02:26.283510 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 10:02:26.283577 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:02:26.283707 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4227,"thread_start_us":4118,"threads_started":1}
I20260924 10:02:26.317663 21535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:26.317910 21535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:26.321097 21535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162dff7be90a42e8b2d9ce74f1042d71: Sending DeleteTablet for 3 replicas of tablet 2d93d3c58f274af4aecc585061ee5a99
I20260924 10:02:26.321864 21675 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93d3c58f274af4aecc585061ee5a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:26 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260924 10:02:26.321893 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93d3c58f274af4aecc585061ee5a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:26 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50924
I20260924 10:02:26.321961 21953 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93d3c58f274af4aecc585061ee5a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:26 UTC) from {username='oryx', principal='oryx/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50848
I20260924 10:02:26.322188 22110 tablet_replica.cc:330] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: stopping tablet replica
I20260924 10:02:26.322242 22112 tablet_replica.cc:330] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: stopping tablet replica
I20260924 10:02:26.322387 22110 raft_consensus.cc:2229] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:26.322418 22112 raft_consensus.cc:2229] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:26.322606 22112 raft_consensus.cc:2258] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:26.322705 22111 tablet_replica.cc:330] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: stopping tablet replica
I20260924 10:02:26.322857 22111 raft_consensus.cc:2229] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:26.323045 22111 raft_consensus.cc:2258] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:26.323592 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 21609
I20260924 10:02:26.324276 22111 ts_tablet_manager.cc:1905] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:26.325953 22112 ts_tablet_manager.cc:1905] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:26.327225 22110 raft_consensus.cc:2258] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:26.328465 22110 ts_tablet_manager.cc:1905] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:26.332767 22112 ts_tablet_manager.cc:1918] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:02:26.332916 22112 log.cc:1196] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/2d93d3c58f274af4aecc585061ee5a99
I20260924 10:02:26.333386 22112 ts_tablet_manager.cc:1939] T 2d93d3c58f274af4aecc585061ee5a99 P 918efb90e64340fc9b848757541e4f19: Deleting consensus metadata
I20260924 10:02:26.334487 21525 catalog_manager.cc:4841] TS 918efb90e64340fc9b848757541e4f19 (127.13.162.3:35171): tablet 2d93d3c58f274af4aecc585061ee5a99 (table test-table [id=c4056cbb6cb04510b4fd4012e224c856]) successfully deleted
I20260924 10:02:26.336849 22111 ts_tablet_manager.cc:1918] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:02:26.336984 22111 log.cc:1196] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/2d93d3c58f274af4aecc585061ee5a99
I20260924 10:02:26.337386 22111 ts_tablet_manager.cc:1939] T 2d93d3c58f274af4aecc585061ee5a99 P 42cd00c6a5a744b4b541c5d738a2d7c6: Deleting consensus metadata
I20260924 10:02:26.338483 21524 catalog_manager.cc:4841] TS 42cd00c6a5a744b4b541c5d738a2d7c6 (127.13.162.2:43077): tablet 2d93d3c58f274af4aecc585061ee5a99 (table test-table [id=c4056cbb6cb04510b4fd4012e224c856]) successfully deleted
I20260924 10:02:26.340340 22110 ts_tablet_manager.cc:1918] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:02:26.340482 22110 log.cc:1196] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/2d93d3c58f274af4aecc585061ee5a99
I20260924 10:02:26.340905 22110 ts_tablet_manager.cc:1939] T 2d93d3c58f274af4aecc585061ee5a99 P b9546c0961e644fc829a06a27dae4fd4: Deleting consensus metadata
I20260924 10:02:26.341979 21524 catalog_manager.cc:4841] TS b9546c0961e644fc829a06a27dae4fd4 (127.13.162.1:37691): tablet 2d93d3c58f274af4aecc585061ee5a99 (table test-table [id=c4056cbb6cb04510b4fd4012e224c856]) successfully deleted
Sep 24 10:02:26 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.1@KRBTEST.COM
I20260924 10:02:26.350726 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.325084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.13.162.1:37691)
0924 10:02:26.325259 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.325267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.334688 (+ 9421us) server_negotiation.cc:407] Connection header received
0924 10:02:26.334959 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.334965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.335051 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.335163 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:26.336698 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.337787 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.339535 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.340226 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.340361 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.344900 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:26.344911 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:26.344927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:26.344959 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:26.348136 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:26.348897 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:26.348904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:26.348906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:26.349012 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:26.349360 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:26.349363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:26.349365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:26.349706 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:26.349969 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.350227 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.350425 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:02:26.351331 21740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:26.470942 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 21609
I20260924 10:02:26.487473 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 21749
Sep 24 10:02:26 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.2@KRBTEST.COM
I20260924 10:02:26.501643 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.488181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.13.162.2:43077)
0924 10:02:26.488331 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.488367 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.488385 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:26.488525 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.488528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.488598 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.488677 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:26.489890 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.490691 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.491899 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.492288 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.492385 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.496556 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:26.496569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:26.496585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:26.496619 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:26.499307 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:26.499952 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:26.499959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:26.499961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:26.500066 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:26.500395 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:26.500398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:26.500400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:26.500751 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:26.500969 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.501220 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.501378 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:02:26.502300 21879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:26.621263 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 21749
I20260924 10:02:26.639058 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 21888
Sep 24 10:02:26 dist-test-slave-jvsj krb5kdc[21493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.162.3@KRBTEST.COM
I20260924 10:02:26.653251 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:26.639778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.13.162.3:35171)
0924 10:02:26.639928 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:26.639932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:26.639948 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:02:26.640064 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:26.640069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:26.640137 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:26.640217 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:26.641419 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.642102 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.643443 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:26.643857 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:26.643948 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:26.647931 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:26.647945 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:26.647961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:26.648010 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:26.650620 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:26.651327 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:26.651337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:26.651340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:26.651459 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:26.651793 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:26.651798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:26.651802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:26.652170 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:26.652443 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:26.652733 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 10:02:26.652936 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:02:26.653798 22018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:26.792248 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 21888
I20260924 10:02:26.811776 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 21506
I20260924 10:02:26.812884 21578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:26.922713 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 21506
2026-09-24T10:02:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6238 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:27 dist-test-slave-jvsj krb5kdc[22162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:27 dist-test-slave-jvsj krb5kdc[22162](info): set up 2 sockets
Sep 24 10:02:27 dist-test-slave-jvsj krb5kdc[22162](info): commencing operation
krb5kdc: starting...
W20260924 10:02:27.930994 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:28 dist-test-slave-jvsj krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:02:28.133816 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:40673
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:40673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:28.448014 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:28.448328 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:28.448453 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:28.455657 22175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:28.455787 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:28.455868 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:28.455941 22175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:28.456005 22175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:28.466790 22175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:40673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:40673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:28.468297 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:28.469604 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:28.481271 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:28.481356 22183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:28.481271 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:28.482465 22175 server_base.cc:1034] running on GCE node
I20260924 10:02:28.483261 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:28.484678 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:28.485882 22175 hybrid_clock.cc:648] HybridClock initialized: now 1790244148485836 us; error 35 us; skew 500 ppm
I20260924 10:02:28.486202 22175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T10:02:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1642 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:28 dist-test-slave-jvsj krb5kdc[22189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:28 dist-test-slave-jvsj krb5kdc[22189](info): set up 2 sockets
Sep 24 10:02:28 dist-test-slave-jvsj krb5kdc[22189](info): commencing operation
krb5kdc: starting...
W20260924 10:02:29.567214 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:29 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:02:29.771152 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42009
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:30.081674 22202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:30.081965 22202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:30.082090 22202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:30.089164 22202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:30.089263 22202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:30.089339 22202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:30.089412 22202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:30.089471 22202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:30.099211 22202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:42009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:30.100479 22202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:30.101753 22202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:30.112993 22207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:30.113065 22208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:30.113171 22210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:30.113731 22202 server_base.cc:1034] running on GCE node
I20260924 10:02:30.114557 22202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:30.115936 22202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:30.117154 22202 hybrid_clock.cc:648] HybridClock initialized: now 1790244150117095 us; error 39 us; skew 500 ppm
I20260924 10:02:30.117466 22202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:30 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244150, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:30.120386 22202 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:02:30.121855 22202 webserver.cc:460] Webserver started at http://127.13.162.62:38425/ using document root <none> and password file <none>
I20260924 10:02:30.122558 22202 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:30.122658 22202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:30.122900 22202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:30.125681 22202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "d04bcc8e33614ceebad957d6645460e6"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "6b8baefef3d92bcae2062d275cc6c520"
server_key_iv: "172d3b1f3181f3cb0f63f777579f35ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:30.126405 22202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "d04bcc8e33614ceebad957d6645460e6"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "6b8baefef3d92bcae2062d275cc6c520"
server_key_iv: "172d3b1f3181f3cb0f63f777579f35ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:30.131372 22202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 10:02:30.134687 22217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:30.136067 22202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:30.136207 22202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "d04bcc8e33614ceebad957d6645460e6"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "6b8baefef3d92bcae2062d275cc6c520"
server_key_iv: "172d3b1f3181f3cb0f63f777579f35ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:30.136363 22202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:30.163877 22202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:30.168114 22202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:30.168346 22202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:30.185491 22202 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:42009
I20260924 10:02:30.185521 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:42009 every 8 connection(s)
I20260924 10:02:30.186818 22202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:30.190608 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:30.190594 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22202
I20260924 10:02:30.190785 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:30.191601 13960 external_mini_cluster.cc:1428] Setting key 41a184d4d9f301e0c82c070d76ecef0a
I20260924 10:02:30.204037 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Bootstrap starting.
Sep 24 10:02:30 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:30.207867 22269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:30.209024 22269 log.cc:824] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:30.212368 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: No bootstrap required, opened a new log
I20260924 10:02:30.215564 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:30.193625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.13.162.62:42009)
0924 10:02:30.194416 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:30.194432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:30.194482 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:02:30.196597 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:30.196623 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:30.197076 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:30.197416 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:30.199126 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:30.200432 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:30.202182 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:30.202953 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:30.203086 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:30.207694 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:30.207709 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:30.207725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:30.207769 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:30.211214 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:30.212063 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:30.212071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:30.212077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:30.212249 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:30.212706 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:30.212712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:30.212715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:30.213297 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:30.213532 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:30.214053 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 10:02:30.214510 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":338,"threads_started":1}
I20260924 10:02:30.220394 22269 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:30.220721 22269 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:30.220814 22269 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04bcc8e33614ceebad957d6645460e6, State: Initialized, Role: FOLLOWER
I20260924 10:02:30.221369 22269 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:30.221575 22269 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:30.221698 22269 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:30.221835 22269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:30.223966 22269 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:30.224432 22269 leader_election.cc:302] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04bcc8e33614ceebad957d6645460e6; no voters:
I20260924 10:02:30.224946 22269 leader_election.cc:288] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:30.225060 22274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:30.225646 22274 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 LEADER]: Becoming Leader. State: Replica: d04bcc8e33614ceebad957d6645460e6, State: Running, Role: LEADER
I20260924 10:02:30.226203 22274 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:30.226855 22269 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:30.229115 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04bcc8e33614ceebad957d6645460e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } } }
I20260924 10:02:30.229507 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:30.229133 22276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04bcc8e33614ceebad957d6645460e6. Latest consensus state: current_term: 1 leader_uuid: "d04bcc8e33614ceebad957d6645460e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } } }
I20260924 10:02:30.231896 22276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:30.232488 22280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:30.237672 22280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:30.246562 22280 catalog_manager.cc:1347] Generated new cluster ID: ede799ccecbe41f49fb3905ca626aa3e
I20260924 10:02:30.246693 22280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:30.279997 22280 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:30.281369 22280 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:30.291168 22280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Generated new TSK 0
I20260924 10:02:30.292075 22280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:02:30.355427 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42009
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:30.675207 22295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:30.675482 22295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:30.675623 22295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:30.682888 22295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:30.683002 22295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:30.683157 22295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:30.692910 22295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:30.694228 22295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:30.695640 22295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:30.708827 22300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:30.708755 22301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:30.708774 22303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:30.709682 22295 server_base.cc:1034] running on GCE node
I20260924 10:02:30.710292 22295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:30.711315 22295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:30.712551 22295 hybrid_clock.cc:648] HybridClock initialized: now 1790244150712491 us; error 47 us; skew 500 ppm
I20260924 10:02:30.712899 22295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:30 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244150, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:30.716053 22295 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:02:30.717486 22295 webserver.cc:460] Webserver started at http://127.13.162.1:38533/ using document root <none> and password file <none>
I20260924 10:02:30.718199 22295 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:30.718308 22295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:30.718629 22295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:30.721488 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "e120b1135c6943bfa9d9361fecb92a91"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "f9219323b96f77e025f4fbabee5bfc6a"
server_key_iv: "8512c1f8b314ef17ddd874ea99b1f0fb"
server_key_version: "encryptionkey@0"
I20260924 10:02:30.722281 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "e120b1135c6943bfa9d9361fecb92a91"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "f9219323b96f77e025f4fbabee5bfc6a"
server_key_iv: "8512c1f8b314ef17ddd874ea99b1f0fb"
server_key_version: "encryptionkey@0"
I20260924 10:02:30.727509 22295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:02:30.731076 22310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:30.732486 22295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:02:30.732631 22295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "e120b1135c6943bfa9d9361fecb92a91"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "f9219323b96f77e025f4fbabee5bfc6a"
server_key_iv: "8512c1f8b314ef17ddd874ea99b1f0fb"
server_key_version: "encryptionkey@0"
I20260924 10:02:30.732817 22295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:30.754016 22295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:30.758062 22295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:30.758368 22295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:30.759343 22295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:30.760625 22295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:30.760703 22295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:30.760807 22295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:30.760857 22295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:30.790827 22295 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41843
I20260924 10:02:30.790891 22422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41843 every 8 connection(s)
I20260924 10:02:30.792073 22295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:30.795460 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22295
I20260924 10:02:30.795665 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:30.796370 13960 external_mini_cluster.cc:1428] Setting key d30bb90993455dca0fded181c471d640
Sep 24 10:02:30 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244150, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:30.811282 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:30.794433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:48735 (local address 127.13.162.62:42009)
0924 10:02:30.794827 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:30.794832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:30.795872 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 10:02:30.797884 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:30.797888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:30.797962 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:30.798062 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:30.799554 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:30.800253 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:30.801754 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:30.802136 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:30.802243 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:30.805762 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:30.805767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:30.805770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:30.805802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:30.808113 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:30.808808 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:30.808813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:30.808817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:30.808957 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:30.809460 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:30.809465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:30.809468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:30.809820 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:30.809953 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:30.810686 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 10:02:30.810904 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":194,"threads_started":1}
I20260924 10:02:30.812341 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:30.794925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:48735)
0924 10:02:30.795702 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:30.795758 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:30.797629 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:30.798367 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:30.798378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:30.798856 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:30.799403 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:30.799419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:30.800394 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:30.800398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:30.801616 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:30.801627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:30.802367 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:30.802375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:30.802603 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:30.803196 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:30.803227 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:30.805544 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:30.808256 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:30.808265 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:30.808289 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:30.808684 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:30.809086 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:30.809091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:30.809095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:30.809331 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:30.809934 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:30.809942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:30.810291 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 10:02:30.810570 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 10:02:30.811113 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":322,"threads_started":1}
I20260924 10:02:30.814023 22423 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42009
I20260924 10:02:30.814440 22423 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:30.815174 22423 heartbeater.cc:511] Master 127.13.162.62:42009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:02:30.817297 22234 ts_manager.cc:194] Registered new tserver with Master: e120b1135c6943bfa9d9361fecb92a91 (127.13.162.1:41843)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:02:30.819164 22234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:48735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:02:30.841656 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42009
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:31.159574 22429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:31.159868 22429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:31.159999 22429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:31.167452 22429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:31.167579 22429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:31.167739 22429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:31.177863 22429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:31.179234 22429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:31.180508 22429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:31.193413 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:31.193544 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:31.193391 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:31.194222 22429 server_base.cc:1034] running on GCE node
I20260924 10:02:31.194927 22429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:31.196017 22429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:31.197240 22429 hybrid_clock.cc:648] HybridClock initialized: now 1790244151197175 us; error 52 us; skew 500 ppm
I20260924 10:02:31.197593 22429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:31 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244151, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:31.201329 22429 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:02:31.202910 22429 webserver.cc:460] Webserver started at http://127.13.162.2:44911/ using document root <none> and password file <none>
I20260924 10:02:31.203568 22429 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:31.203661 22429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:31.203909 22429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:31.206727 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "3f641d11e3244238991bc1d662c9f2aa"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "4fd2b3e04a61d4f68d9039df81c57f73"
server_key_iv: "495651f0b226a957b605dd631711d664"
server_key_version: "encryptionkey@0"
I20260924 10:02:31.207446 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "3f641d11e3244238991bc1d662c9f2aa"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "4fd2b3e04a61d4f68d9039df81c57f73"
server_key_iv: "495651f0b226a957b605dd631711d664"
server_key_version: "encryptionkey@0"
I20260924 10:02:31.212385 22429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:02:31.215934 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:31.217334 22429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:02:31.217468 22429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "3f641d11e3244238991bc1d662c9f2aa"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "4fd2b3e04a61d4f68d9039df81c57f73"
server_key_iv: "495651f0b226a957b605dd631711d664"
server_key_version: "encryptionkey@0"
I20260924 10:02:31.217631 22429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:31.236199 22429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:31.240527 22429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:31.240763 22429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:31.241727 22429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:31.243144 22429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:31.243214 22429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:31.243317 22429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:31.243366 22429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:31.272311 22429 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:36269
I20260924 10:02:31.272395 22556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:36269 every 8 connection(s)
I20260924 10:02:31.273548 22429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:31.281567 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22429
I20260924 10:02:31.281738 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:31.282297 13960 external_mini_cluster.cc:1428] Setting key 65f899ca604bfedca7ba13f5abef5559
Sep 24 10:02:31 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244151, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:31.294872 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:31.275984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:48033 (local address 127.13.162.62:42009)
0924 10:02:31.276199 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:31.276205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:31.277329 (+ 1124us) server_negotiation.cc:407] Connection header received
0924 10:02:31.279579 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:31.279584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:31.279654 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:31.279745 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:31.281397 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.282478 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.284198 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.284827 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.284961 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:31.288405 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:31.288411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:31.288416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:31.288454 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:31.291645 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:31.292540 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:31.292544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:31.292547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:31.292678 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:31.293140 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:31.293144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:31.293146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:31.293442 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:31.293585 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:31.294427 (+ 842us) server_negotiation.cc:299] Negotiation successful
0924 10:02:31.294599 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:02:31.296181 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:31.276461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:48033)
0924 10:02:31.277170 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:31.277208 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:31.279304 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:31.279984 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:31.279993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:31.280583 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:31.281156 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:31.281172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.282631 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.282636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:31.284038 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:31.284052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.285031 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.285036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:31.285229 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:31.285885 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:31.285912 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:31.288150 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:31.291814 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:31.291823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:31.291843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:31.292413 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:31.292775 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:31.292779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:31.292783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:31.293013 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:31.293562 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:31.293570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:31.293993 (+ 423us) client_negotiation.cc:769] Sending connection context
0924 10:02:31.294282 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 10:02:31.294830 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":296,"threads_started":1}
I20260924 10:02:31.297679 22557 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42009
I20260924 10:02:31.298007 22557 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:31.298800 22557 heartbeater.cc:511] Master 127.13.162.62:42009 requested a full tablet report, sending...
I20260924 10:02:31.300199 22234 ts_manager.cc:194] Registered new tserver with Master: 3f641d11e3244238991bc1d662c9f2aa (127.13.162.2:36269)
I20260924 10:02:31.301002 22234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:48033
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:02:31.328866 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42009
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:31.648270 22562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:31.648553 22562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:31.648651 22562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:31.655948 22562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:31.656067 22562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:31.656188 22562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:31.666373 22562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:31.667743 22562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:31.669013 22562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:31.681432 22568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:31.681456 22567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:31.682057 22570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:31.682119 22562 server_base.cc:1034] running on GCE node
I20260924 10:02:31.682751 22562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:31.683670 22562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:31.684862 22562 hybrid_clock.cc:648] HybridClock initialized: now 1790244151684818 us; error 37 us; skew 500 ppm
I20260924 10:02:31.685159 22562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:31 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244151, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:31.688117 22562 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:02:31.689519 22562 webserver.cc:460] Webserver started at http://127.13.162.3:45895/ using document root <none> and password file <none>
I20260924 10:02:31.690177 22562 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:31.690284 22562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:31.690621 22562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:31.693431 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "beaf89fcb75a42dc83728f12a2b905e0"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "6f7092a5e38569cc040deeebb079cb9a"
server_key_iv: "bd3dc05b0f258b1616ecf664451949ee"
server_key_version: "encryptionkey@0"
I20260924 10:02:31.694139 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "beaf89fcb75a42dc83728f12a2b905e0"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "6f7092a5e38569cc040deeebb079cb9a"
server_key_iv: "bd3dc05b0f258b1616ecf664451949ee"
server_key_version: "encryptionkey@0"
I20260924 10:02:31.699186 22562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 10:02:31.702745 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:31.704088 22562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:02:31.704231 22562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "beaf89fcb75a42dc83728f12a2b905e0"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "6f7092a5e38569cc040deeebb079cb9a"
server_key_iv: "bd3dc05b0f258b1616ecf664451949ee"
server_key_version: "encryptionkey@0"
I20260924 10:02:31.704409 22562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:31.715499 22562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:31.719558 22562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:31.719812 22562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:31.720737 22562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:31.722107 22562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:31.722187 22562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:31.722304 22562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:31.722401 22562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 10:02:31.755474 22562 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:34607
I20260924 10:02:31.755508 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:34607 every 8 connection(s)
I20260924 10:02:31.756732 22562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:31.758663 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22562
I20260924 10:02:31.758812 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:31.759323 13960 external_mini_cluster.cc:1428] Setting key 455ab88fc9af43e62e27c4c19a53e1b0
Sep 24 10:02:31 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244151, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:31.776288 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:31.759200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:55431 (local address 127.13.162.62:42009)
0924 10:02:31.759398 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:31.759404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:31.760557 (+ 1153us) server_negotiation.cc:407] Connection header received
0924 10:02:31.762760 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:31.762764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:31.762830 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:31.762919 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:31.764483 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.765255 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.766875 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.767260 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.767367 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:31.770873 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:31.770878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:31.770881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:31.770918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:31.773285 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:31.774100 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:31.774104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:31.774107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:31.774258 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:31.774803 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:31.774806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:31.774808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:31.775107 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:31.775208 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:31.775889 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 10:02:31.776048 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:02:31.777216 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:31.759694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:55431)
0924 10:02:31.760408 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:31.760461 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:31.762506 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:31.763177 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:31.763185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:31.763701 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:31.764295 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:31.764313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.765415 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.765419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:31.766740 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:31.766757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.767444 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.767448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:31.767655 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:31.768276 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:31.768298 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:31.770625 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:31.773429 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:31.773438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:31.773456 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:31.773972 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:31.774388 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:31.774392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:31.774395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:31.774692 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:31.775213 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:31.775218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:31.775545 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 10:02:31.775771 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 10:02:31.776167 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260924 10:02:31.778669 22690 heartbeater.cc:347] Connected to a master server at 127.13.162.62:42009
I20260924 10:02:31.778967 22690 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:31.779650 22690 heartbeater.cc:511] Master 127.13.162.62:42009 requested a full tablet report, sending...
I20260924 10:02:31.781069 22234 ts_manager.cc:194] Registered new tserver with Master: beaf89fcb75a42dc83728f12a2b905e0 (127.13.162.3:34607)
I20260924 10:02:31.781901 22234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:55431
I20260924 10:02:31.787060 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.1@KRBTEST.COM" created.
I20260924 10:02:31.822432 22423 heartbeater.cc:503] Master 127.13.162.62:42009 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab.
Entry for principal oryx/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab.
WARNING: no policy specified for oryx/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab.
Entry for principal oryx/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab.
WARNING: no policy specified for oryx/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab.
Entry for principal oryx/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab.
I20260924 10:02:31.908685 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22295
Sep 24 10:02:31 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:02:31.924108 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:31.909797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.13.162.1:41843)
0924 10:02:31.910538 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:31.910546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:31.910570 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:02:31.910630 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:31.910636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:31.910837 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:31.910991 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:31.912275 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.913252 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.914511 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:31.915062 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:31.915171 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:31.918904 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:31.918919 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:31.918937 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:31.918974 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:31.921562 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:31.922182 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:31.922188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:31.922190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:31.922307 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:31.922661 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:31.922666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:31.922669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:31.922982 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:31.923240 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:31.923543 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 10:02:31.923780 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":375,"threads_started":1}
I20260924 10:02:31.925359 22418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:32.006680 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22295
I20260924 10:02:32.019814 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22429
Sep 24 10:02:32 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:02:32.034636 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:32.020696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.13.162.2:36269)
0924 10:02:32.021155 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:32.021162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:32.021187 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:02:32.021243 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:32.021248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:32.021473 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:32.021604 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:32.022854 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:32.023842 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:32.024970 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:32.025523 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:32.025622 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:32.029590 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:32.029603 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:32.029618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:32.029654 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:32.032301 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:32.032906 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:32.032913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:32.032915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:32.033034 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:32.033304 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:32.033306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:32.033308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:32.033627 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:32.033848 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:32.034058 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 10:02:32.034265 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260924 10:02:32.035930 22552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:32.117203 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22429
I20260924 10:02:32.129715 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22562
Sep 24 10:02:32 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:02:32.144979 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:32.130627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.13.162.3:34607)
0924 10:02:32.131262 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:32.131269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:32.131297 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:02:32.131353 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:32.131358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:32.131560 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:32.131714 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:32.132953 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:32.133930 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:32.135159 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:32.135696 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:32.135791 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:32.139816 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:32.139829 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:32.139845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:32.139878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:32.142596 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:32.143219 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:32.143226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:32.143228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:32.143347 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:32.143658 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:32.143662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:32.143664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:32.143986 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:32.144226 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:32.144446 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:02:32.144644 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":324,"threads_started":1}
I20260924 10:02:32.146215 22685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:32.227761 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22562
I20260924 10:02:32.240006 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22202
I20260924 10:02:32.241130 22264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:32.340857 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22202
I20260924 10:02:32.357170 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42009
--webserver_interface=127.13.162.62
--webserver_port=38425
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:32.672127 22707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:32.672401 22707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:32.672514 22707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:32.679589 22707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:32.679697 22707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:32.679746 22707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:32.679821 22707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:32.679873 22707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:32.689455 22707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:42009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=38425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:32.690771 22707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:32.692032 22707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:32.703847 22712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:32.703737 22713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:32.703791 22715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:32.704803 22707 server_base.cc:1034] running on GCE node
I20260924 10:02:32.705650 22707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:32.707403 22707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:32.708709 22707 hybrid_clock.cc:648] HybridClock initialized: now 1790244152708609 us; error 89 us; skew 500 ppm
I20260924 10:02:32.709048 22707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:32 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244152, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:32.712268 22707 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:02:32.713887 22707 webserver.cc:460] Webserver started at http://127.13.162.62:38425/ using document root <none> and password file <none>
I20260924 10:02:32.714694 22707 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:32.714838 22707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:32.720549 22707 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:02:32.723541 22722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:32.725104 22707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:02:32.725270 22707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "d04bcc8e33614ceebad957d6645460e6"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "6b8baefef3d92bcae2062d275cc6c520"
server_key_iv: "172d3b1f3181f3cb0f63f777579f35ed"
server_key_version: "encryptionkey@0"
I20260924 10:02:32.726253 22707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:32.742384 22707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:32.746640 22707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:32.746903 22707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:32.763657 22707 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:42009
I20260924 10:02:32.763675 22773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:42009 every 8 connection(s)
I20260924 10:02:32.765128 22707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:32.766188 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22707
I20260924 10:02:32.767019 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:41843
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:42009
--webserver_port=38533
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:02:32.773164 22774 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:02:32.777875 22774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Bootstrap starting.
I20260924 10:02:32.805917 22774 log.cc:824] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:32.817077 22774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:02:32.817721 22774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Bootstrap complete.
I20260924 10:02:32.831470 22774 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:32.832880 22774 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04bcc8e33614ceebad957d6645460e6, State: Initialized, Role: FOLLOWER
I20260924 10:02:32.833760 22774 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:32.834079 22774 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:32.834223 22774 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:32.834472 22774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:02:32.840171 22774 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:32.840795 22774 leader_election.cc:302] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04bcc8e33614ceebad957d6645460e6; no voters:
I20260924 10:02:32.841403 22774 leader_election.cc:288] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:02:32.841542 22778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:02:32.842640 22778 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [term 2 LEADER]: Becoming Leader. State: Replica: d04bcc8e33614ceebad957d6645460e6, State: Running, Role: LEADER
I20260924 10:02:32.843209 22778 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } }
I20260924 10:02:32.843514 22774 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:32.845530 22780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04bcc8e33614ceebad957d6645460e6. Latest consensus state: current_term: 2 leader_uuid: "d04bcc8e33614ceebad957d6645460e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } } }
I20260924 10:02:32.845906 22780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:32.845525 22779 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d04bcc8e33614ceebad957d6645460e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04bcc8e33614ceebad957d6645460e6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42009 } } }
I20260924 10:02:32.846002 22779 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:32.856189 22784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:32.862003 22784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:32.863463 22784 catalog_manager.cc:1259] Loaded cluster ID: ede799ccecbe41f49fb3905ca626aa3e
I20260924 10:02:32.863615 22784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:32.867705 22784 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:32.869261 22784 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d04bcc8e33614ceebad957d6645460e6: Loaded TSK: 0
I20260924 10:02:32.870929 22784 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:02:33.125054 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:33.125345 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:33.125469 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:33.132874 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:33.132987 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:33.133128 22776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:33.143229 22776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.1.keytab
--principal=oryx/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:41843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=38533
--tserver_master_addrs=127.13.162.62:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:33.144614 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:33.145923 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:33.164958 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:33.165551 22803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:33.165653 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:33.166636 22776 server_base.cc:1034] running on GCE node
I20260924 10:02:33.167273 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:33.168308 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:33.169531 22776 hybrid_clock.cc:648] HybridClock initialized: now 1790244153169474 us; error 53 us; skew 500 ppm
I20260924 10:02:33.169848 22776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:33 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244153, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:33.172940 22776 init.cc:377] Logged in from keytab as oryx/127.13.162.1@KRBTEST.COM (short username oryx)
I20260924 10:02:33.174469 22776 webserver.cc:460] Webserver started at http://127.13.162.1:38533/ using document root <none> and password file <none>
I20260924 10:02:33.175151 22776 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:33.175274 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:33.180610 22776 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 10:02:33.183291 22810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:33.184630 22776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:02:33.184772 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "e120b1135c6943bfa9d9361fecb92a91"
format_stamp: "Formatted at 2026-09-24 10:02:30 on dist-test-slave-jvsj"
server_key: "f9219323b96f77e025f4fbabee5bfc6a"
server_key_iv: "8512c1f8b314ef17ddd874ea99b1f0fb"
server_key_version: "encryptionkey@0"
I20260924 10:02:33.185684 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:33.214088 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:33.218171 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:33.218439 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:33.219434 22776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:33.220775 22776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:33.220856 22776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:33.220961 22776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:33.221041 22776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:33.251080 22776 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41843
I20260924 10:02:33.251137 22922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41843 every 8 connection(s)
I20260924 10:02:33.252544 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:33.262220 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22776
I20260924 10:02:33.262826 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:36269
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:42009
--webserver_port=44911
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:02:33 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:33 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:02:33.588168 22927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:33.588478 22927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:33.588618 22927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:33.596560 22927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:33.596698 22927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:33.596868 22927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:33.607110 22927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.2.keytab
--principal=oryx/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:36269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=44911
--tserver_master_addrs=127.13.162.62:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:33.608501 22927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:33.609897 22927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:33.623956 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:33.624401 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:33.624532 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:33.624847 22927 server_base.cc:1034] running on GCE node
I20260924 10:02:33.625588 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:33.626653 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:33.627889 22927 hybrid_clock.cc:648] HybridClock initialized: now 1790244153627840 us; error 42 us; skew 500 ppm
I20260924 10:02:33.628216 22927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:33 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244153, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:33.636603 22927 init.cc:377] Logged in from keytab as oryx/127.13.162.2@KRBTEST.COM (short username oryx)
I20260924 10:02:33.638296 22927 webserver.cc:460] Webserver started at http://127.13.162.2:44911/ using document root <none> and password file <none>
I20260924 10:02:33.639102 22927 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:33.639232 22927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:33.644958 22927 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 10:02:33.647830 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:33.649273 22927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:33.649433 22927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "3f641d11e3244238991bc1d662c9f2aa"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "4fd2b3e04a61d4f68d9039df81c57f73"
server_key_iv: "495651f0b226a957b605dd631711d664"
server_key_version: "encryptionkey@0"
I20260924 10:02:33.650701 22927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:33.674009 22927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:33.678517 22927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:33.678777 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:33.679819 22927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:33.681289 22927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:33.681388 22927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:33.681496 22927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:33.681563 22927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:33.718209 22927 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:36269
I20260924 10:02:33.718261 23055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:36269 every 8 connection(s)
I20260924 10:02:33.719733 22927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:33.726127 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 22927
I20260924 10:02:33.726843 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:34607
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:42009
--webserver_port=45895
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:02:33 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:33 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:02:34.262238 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:34.262710 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:34.262882 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:34.275170 22926 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.009s user 0.008s sys 0.007s
W20260924 10:02:34.276202 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:34.276446 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:34.276741 23060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:34.277063 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:33.255574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:44467)
0924 10:02:33.256388 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:33.256433 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:33.258249 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:33.259785 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:33.259798 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:33.260400 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:33.260949 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:33.260973 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.262674 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.262677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:33.263976 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:33.263989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.264993 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.265000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:33.265246 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:33.265877 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:33.265901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:34.275940 (+1010039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
W20260924 10:02:34.277063 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:33.255512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:44467 (local address 127.13.162.62:42009)
0924 10:02:33.256311 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:33.256329 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:33.256555 (+ 226us) server_negotiation.cc:407] Connection header received
0924 10:02:33.258886 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:33.258902 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:33.259266 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:33.259616 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:33.261168 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.262521 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.264112 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.264768 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.264949 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.276324 (+1011375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:44467: BlockingRecv error: recv got EOF from 127.13.162.1:44467 (error 108)
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":280,"threads_started":1}
W20260924 10:02:34.278285 22923 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:42009 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.162.62:42009: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:34.287283 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:34.278959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:46983)
0924 10:02:34.279212 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:34.279230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:34.279345 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:34.279637 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:34.279642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:34.279999 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:34.280156 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.280165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.281126 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.281131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.282068 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.282087 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.282810 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.282817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.282958 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.283499 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:34.283515 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:34.287000 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260924 10:02:34.287400 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:34.279050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:46983 (local address 127.13.162.62:42009)
0924 10:02:34.279207 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:34.279213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:34.279282 (+ 69us) server_negotiation.cc:407] Connection header received
0924 10:02:34.279454 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:34.279460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:34.279524 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:34.279607 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:34.280285 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.280990 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.282201 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.282648 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.282751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.287199 (+ 4448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:46983: BlockingRecv error: recv got EOF from 127.13.162.1:46983 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:34.294514 23060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/krb5kdc/oryx_127.13.162.3.keytab
--principal=oryx/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:34607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=45895
--tserver_master_addrs=127.13.162.62:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:34.295956 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:34.288085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:39811)
0924 10:02:34.288335 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:34.288352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:34.288455 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:34.288797 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:34.288802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:34.289112 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:34.289250 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.289259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.290362 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.290367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.291278 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.291288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.291931 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.291937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.292110 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.292599 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:34.292615 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:34.295697 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260924 10:02:34.296092 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:34.288190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:39811 (local address 127.13.162.62:42009)
0924 10:02:34.288327 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:34.288332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:34.288412 (+ 80us) server_negotiation.cc:407] Connection header received
0924 10:02:34.288629 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:34.288632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:34.288696 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:34.288776 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:34.289414 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.290207 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.291399 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.291805 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.291905 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.295900 (+ 3995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:39811: BlockingRecv error: recv got EOF from 127.13.162.1:39811 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:02:34.296510 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260924 10:02:34.296553 22923 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:02:34.298516 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:34.314601 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:34.314566 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:34.315873 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:34.316157 23060 server_base.cc:1034] running on GCE node
I20260924 10:02:34.316972 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:34.318311 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:34.319593 23060 hybrid_clock.cc:648] HybridClock initialized: now 1790244154319525 us; error 48 us; skew 500 ppm
I20260924 10:02:34.320025 23060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244154, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:34.324550 23060 init.cc:377] Logged in from keytab as oryx/127.13.162.3@KRBTEST.COM (short username oryx)
I20260924 10:02:34.326650 23060 webserver.cc:460] Webserver started at http://127.13.162.3:45895/ using document root <none> and password file <none>
I20260924 10:02:34.327685 23060 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:34.327849 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:34.335937 23060 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:02:34.339680 23076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:34.341585 23060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:02:34.341776 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "beaf89fcb75a42dc83728f12a2b905e0"
format_stamp: "Formatted at 2026-09-24 10:02:31 on dist-test-slave-jvsj"
server_key: "6f7092a5e38569cc040deeebb079cb9a"
server_key_iv: "bd3dc05b0f258b1616ecf664451949ee"
server_key_version: "encryptionkey@0"
I20260924 10:02:34.343369 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:34.369122 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:34.375168 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:34.375520 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:34.376837 23060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:34.378858 23060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:34.378996 23060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:34.379117 23060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:34.379211 23060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:34.414273 23060 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:34607
I20260924 10:02:34.414273 23188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:34607 every 8 connection(s)
I20260924 10:02:34.416460 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:34.416922 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23060
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:34.447469 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:34.418389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.13.162.62:42009)
0924 10:02:34.418566 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:34.418572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:34.431006 (+ 12434us) server_negotiation.cc:407] Connection header received
0924 10:02:34.431401 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:34.431408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:34.431524 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:34.431659 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:34.433526 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.434243 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.435997 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.436385 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.436484 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.439223 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:34.439240 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:34.439267 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:34.439321 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:34.443477 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:34.444399 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:34.444410 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:34.444422 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:34.444684 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:34.445054 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:34.445059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:34.445063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:34.445792 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:34.446114 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:34.446595 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 10:02:34.447022 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":31488}
W20260924 10:02:34.903674 23059 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.169s user 0.007s sys 0.013s
W20260924 10:02:34.904942 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:33.722610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:45487 (local address 127.13.162.62:42009)
0924 10:02:33.723046 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:33.723051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:33.724362 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 10:02:33.726536 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:33.726553 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:33.726660 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:33.726787 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:33.728775 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.730053 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.732268 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.733003 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.733154 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.904638 (+1171484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:45487: BlockingRecv error: recv got EOF from 127.13.162.2:45487 (error 108)
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
I20260924 10:02:34.905489 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:33.723381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:45487)
0924 10:02:33.724196 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:33.724242 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:33.726136 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:33.727089 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:33.727103 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:33.727779 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:33.728578 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:33.728602 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.730230 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.730235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:33.732105 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:33.732120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:33.733575 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:33.733584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:33.733961 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:33.734850 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:33.734893 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:34.904443 (+1169550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":324,"threads_started":1}
W20260924 10:02:34.906365 23056 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:42009 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.162.62:42009: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:34.915216 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:34.907028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:52023)
0924 10:02:34.907226 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:34.907246 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:34.907359 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:34.907652 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:34.907657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:34.908016 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:34.908161 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.908170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.909121 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.909127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.910082 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.910093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.910909 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.910915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.911060 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.911594 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:34.911610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:34.914897 (+ 3287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 10:02:34.915305 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:34.907091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:52023 (local address 127.13.162.62:42009)
0924 10:02:34.907237 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:34.907297 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:34.907318 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:02:34.907475 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:34.907481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:34.907540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:34.907616 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:34.908293 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.908996 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.910209 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.910754 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.910858 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.915077 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:52023: BlockingRecv error: recv got EOF from 127.13.162.2:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:34 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:34.924451 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:34.916112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:40971)
0924 10:02:34.916265 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:34.916282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:34.916367 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:34.916805 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:34.916808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:34.917069 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:34.917207 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.917214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.918752 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.918756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.919721 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.919733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.920641 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.920645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.920772 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.921299 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:34.921316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:34.924270 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:02:34.924760 23056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:02:34.925179 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:34.916346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:40971 (local address 127.13.162.62:42009)
0924 10:02:34.916523 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:34.916530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:34.916548 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:34.916599 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:34.916605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:34.916679 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:34.916786 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:34.917342 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.918593 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.919839 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.920502 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.920626 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.924519 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:40971: BlockingRecv error: recv got EOF from 127.13.162.2:40971 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:02:35.308525 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:35.297260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:43929 (local address 127.13.162.62:42009)
0924 10:02:35.297464 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:35.297471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:35.297822 (+ 351us) server_negotiation.cc:407] Connection header received
0924 10:02:35.297982 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:35.297988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:35.298097 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:35.298216 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:35.298965 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.300234 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.302343 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.302865 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.302965 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.308359 (+ 5394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:43929: BlockingRecv error: recv got EOF from 127.13.162.1:43929 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:02:35.308565 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:35.297119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:43929)
0924 10:02:35.297723 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:35.297744 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:35.297869 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:35.298263 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:35.298266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:35.298672 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:35.298838 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.298847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.300437 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.300444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.302158 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.302175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.303045 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.303052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.303285 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.303948 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:35.303973 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:35.308196 (+ 4223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":372,"threads_started":1}
W20260924 10:02:35.481675 23192 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.041s user 0.001s sys 0.012s
W20260924 10:02:35.482975 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:34.421025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:40795 (local address 127.13.162.62:42009)
0924 10:02:34.431623 (+ 10598us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:34.431629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:34.431645 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:02:34.431691 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:34.431695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:34.431794 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:34.431898 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:34.434661 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.435793 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.438450 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.439028 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.439137 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.482692 (+1043555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:40795: BlockingRecv error: recv got EOF from 127.13.162.3:40795 (error 108)
Metrics: {"server-negotiator.queue_time_us":10495,"spinlock_wait_cycles":3072,"thread_start_us":10368,"threads_started":1}
I20260924 10:02:35.483588 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:34.420119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:40795)
0924 10:02:34.421658 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:34.421723 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:34.424424 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:34.432307 (+ 7883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:34.432323 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:34.433359 (+ 1036us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:34.434459 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.434492 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.435941 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.435947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.438221 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:34.438238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:34.439273 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:34.439280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:34.439508 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:34.440130 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:34.440151 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:35.482498 (+1042347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1161,"thread_start_us":925,"threads_started":1}
W20260924 10:02:35.484480 23189 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:42009 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.162.62:42009: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:35.493005 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:35.485053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:48569)
0924 10:02:35.485198 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:35.485214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:35.485328 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:35.485632 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:35.485637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:35.486002 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:35.486155 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.486164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.487235 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.487239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.488208 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.488220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.488939 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.488943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.489063 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.489627 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:35.489644 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:35.492805 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:02:35.493218 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:35.485128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:48569 (local address 127.13.162.62:42009)
0924 10:02:35.485379 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:35.485385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:35.485404 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:02:35.485451 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:35.485456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:35.485516 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:35.485590 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:35.486285 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.487090 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.488324 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.488803 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.488898 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.493010 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:48569: BlockingRecv error: recv got EOF from 127.13.162.3:48569 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:35.500908 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:35.493629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:43749)
0924 10:02:35.493766 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:35.493783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:35.493863 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:35.494139 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:35.494143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:35.494505 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:35.494650 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.494657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.495773 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.495779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.496696 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.496706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.497312 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.497316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.497431 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.497923 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:35.497939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:35.500728 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 10:02:35.501140 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:35.493706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:43749 (local address 127.13.162.62:42009)
0924 10:02:35.493852 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:35.493856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:35.493872 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:02:35.493942 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:35.493946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:35.494012 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:35.494094 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:35.494768 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.495640 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.496805 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.497207 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.497296 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.500896 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:43749: BlockingRecv error: recv got EOF from 127.13.162.3:43749 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 10:02:35.501168 23189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:35 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:35.933779 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:35.925291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:36975)
0924 10:02:35.925825 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:35.925840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:35.925967 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:35.926289 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:35.926294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:35.926734 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:35.926873 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.926882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.927854 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.927858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.928852 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:35.928865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.929578 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.929584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:35.929737 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.930277 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:35.930298 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:35.933551 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":325,"threads_started":1}
W20260924 10:02:35.933923 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:35.925400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:36975 (local address 127.13.162.62:42009)
0924 10:02:35.925585 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:35.925592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:35.925921 (+ 329us) server_negotiation.cc:407] Connection header received
0924 10:02:35.926090 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:35.926094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:35.926165 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:35.926245 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:35.927021 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.927719 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.928976 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:35.929403 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:35.929510 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:35.933739 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:36975: BlockingRecv error: recv got EOF from 127.13.162.2:36975 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:02:36 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:36 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:36.317821 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:36.309512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:52717)
0924 10:02:36.309980 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:36.310000 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:36.310110 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:36.310468 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:36.310473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:36.310919 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:36.311092 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:36.311101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.312081 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.312086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:36.313032 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:36.313046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.313756 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.313762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:36.313913 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:36.314470 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:36.314487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:36.317493 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":270,"threads_started":1}
W20260924 10:02:36.317902 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:36.309544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52717 (local address 127.13.162.62:42009)
0924 10:02:36.309725 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:36.309731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:36.310065 (+ 334us) server_negotiation.cc:407] Connection header received
0924 10:02:36.310232 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:36.310237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:36.310306 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:36.310416 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:36.311229 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.311934 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.313158 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.313603 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.313705 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:36.317692 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:52717: BlockingRecv error: recv got EOF from 127.13.162.1:52717 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:02:36 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:36 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:36.510442 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:36.501864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:60335)
0924 10:02:36.502387 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:36.502403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:36.502510 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:36.502834 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:36.502837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:36.503335 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:36.503479 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:36.503488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.504428 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.504431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:36.505352 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:36.505363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.506092 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.506098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:36.506252 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:36.506850 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:36.506867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:36.510095 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
W20260924 10:02:36.510563 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:36.501939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:60335 (local address 127.13.162.62:42009)
0924 10:02:36.502143 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:36.502149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:36.502478 (+ 329us) server_negotiation.cc:407] Connection header received
0924 10:02:36.502615 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:36.502621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:36.502703 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:36.502790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:36.503600 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.504324 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.505479 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.505930 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.506021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:36.510350 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:60335: BlockingRecv error: recv got EOF from 127.13.162.3:60335 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 10:02:36 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:36 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:36.943663 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:36.934748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:33419)
0924 10:02:36.935276 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:36.935295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:36.935437 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:36.935801 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:36.935805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:36.936338 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:36.936569 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:36.936582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.937529 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.937533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:36.938469 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:36.938480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.939190 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.939195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:36.939318 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:36.939876 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:36.939894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:36.943423 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":279,"threads_started":1}
W20260924 10:02:36.943843 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:36.934844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:33419 (local address 127.13.162.62:42009)
0924 10:02:36.935020 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:36.935026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:36.935366 (+ 340us) server_negotiation.cc:407] Connection header received
0924 10:02:36.935570 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:36.935576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:36.935655 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:36.935737 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:36.936699 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.937411 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.938581 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:36.938986 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:36.939076 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:36.943625 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:33419: BlockingRecv error: recv got EOF from 127.13.162.2:33419 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:02:37 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:37 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:37.327310 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:37.318767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:39043)
0924 10:02:37.319255 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:37.319280 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:37.319482 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:37.319793 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:37.319798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:37.320179 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:37.320346 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:37.320357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.321375 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.321381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:37.322278 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:37.322290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.323003 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.323008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:37.323182 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:37.323698 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:37.323715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:37.326989 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":233,"threads_started":1}
W20260924 10:02:37.327441 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:37.319036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:39043 (local address 127.13.162.62:42009)
0924 10:02:37.319268 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:37.319274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:37.319349 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:02:37.319605 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:37.319610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:37.319668 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:37.319744 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:37.320495 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.321234 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.322433 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.322869 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.322977 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:37.327200 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:39043: BlockingRecv error: recv got EOF from 127.13.162.1:39043 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 10:02:37 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:37 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:37.520661 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:37.511430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:35575)
0924 10:02:37.511862 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:37.511882 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:37.512000 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:37.512327 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:37.512330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:37.512710 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:37.512934 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:37.512947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.514213 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.514218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:37.515373 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:37.515387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.516036 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.516041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:37.516161 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:37.516731 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:37.516749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:37.520403 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":237,"threads_started":1}
W20260924 10:02:37.520885 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:37.511473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:35575 (local address 127.13.162.62:42009)
0924 10:02:37.511668 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:37.511675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:37.511960 (+ 285us) server_negotiation.cc:407] Connection header received
0924 10:02:37.512092 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:37.512096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:37.512171 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:37.512252 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:37.513088 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.514077 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.515499 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.515892 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.516015 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:37.520579 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:35575: BlockingRecv error: recv got EOF from 127.13.162.3:35575 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 10:02:37 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:37 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:37.953045 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:37.944637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:46307)
0924 10:02:37.945087 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:37.945102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:37.945216 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:37.945477 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:37.945481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:37.945807 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:37.945991 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:37.946001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.947036 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.947041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:37.947969 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:37.947983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.948592 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.948596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:37.948762 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:37.949391 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:37.949424 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:37.952785 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
W20260924 10:02:37.953135 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:37.944675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:46307 (local address 127.13.162.62:42009)
0924 10:02:37.944855 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:37.944862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:37.945170 (+ 308us) server_negotiation.cc:407] Connection header received
0924 10:02:37.945308 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:37.945311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:37.945370 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:37.945445 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:37.946112 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.946895 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.948091 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:37.948471 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:37.948554 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:37.952957 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:46307: BlockingRecv error: recv got EOF from 127.13.162.2:46307 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:02:38 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:38 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:38.337009 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:38.328381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:51865)
0924 10:02:38.328981 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:38.329044 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:38.329172 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:38.329536 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:38.329542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:38.329973 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:38.330131 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:38.330164 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.331375 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.331379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:38.332337 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:38.332347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.332998 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.333001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:38.333116 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:38.333681 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:38.333698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:38.336769 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":346,"threads_started":1}
W20260924 10:02:38.337239 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:38.328509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51865 (local address 127.13.162.62:42009)
0924 10:02:38.328681 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:38.328686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:38.329140 (+ 454us) server_negotiation.cc:407] Connection header received
0924 10:02:38.329285 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:38.329291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:38.329372 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:38.329463 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:38.330287 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.331225 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.332491 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.332886 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.332978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:38.337007 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:51865: BlockingRecv error: recv got EOF from 127.13.162.1:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 10:02:38 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:38 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:38.530187 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:38.521627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:48407)
0924 10:02:38.522255 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:38.522272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:38.522448 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:38.522824 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:38.522828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:38.523224 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:38.523381 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:38.523390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.524358 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.524361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:38.525289 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:38.525300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.525928 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.525932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:38.526068 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:38.526616 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:38.526633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:38.529956 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":393,"threads_started":1}
W20260924 10:02:38.530391 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:38.521745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:48407 (local address 127.13.162.62:42009)
0924 10:02:38.521942 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:38.521949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:38.522368 (+ 419us) server_negotiation.cc:407] Connection header received
0924 10:02:38.522570 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:38.522575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:38.522689 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:38.522769 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:38.523517 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.524236 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.525412 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.525808 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.525900 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:38.530152 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:48407: BlockingRecv error: recv got EOF from 127.13.162.3:48407 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 10:02:38 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:38 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:38.966734 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:38.953879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:55865)
0924 10:02:38.954368 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:38.954384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:38.954485 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:38.954814 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:38.954818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:38.955159 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:38.955315 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:38.955323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.956490 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.956496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:38.957520 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:38.957534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.958200 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.958204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:38.958357 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:38.963003 (+ 4646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:38.963021 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:38.966472 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
W20260924 10:02:38.966869 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:38.953952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:55865 (local address 127.13.162.62:42009)
0924 10:02:38.954106 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:38.954111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:38.954445 (+ 334us) server_negotiation.cc:407] Connection header received
0924 10:02:38.954583 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:38.954588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:38.954651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:38.954761 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:38.955474 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.956319 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.957666 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:38.958087 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:38.958190 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:38.966660 (+ 8470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:55865: BlockingRecv error: recv got EOF from 127.13.162.2:55865 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:02:39 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:39 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:39.346978 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:39.337996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:49249)
0924 10:02:39.338511 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:39.338531 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:39.338668 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:39.339004 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:39.339008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:39.339454 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:39.339600 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:39.339610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.340746 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.340751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:39.341846 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:39.341859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.342574 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.342580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:39.342758 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:39.343347 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:39.343369 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:39.346636 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":259,"threads_started":1}
W20260924 10:02:39.347035 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:39.338141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:49249 (local address 127.13.162.62:42009)
0924 10:02:39.338391 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:39.338397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:39.338616 (+ 219us) server_negotiation.cc:407] Connection header received
0924 10:02:39.338797 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:39.338802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:39.338868 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:39.338963 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:39.339750 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.340608 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.341974 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.342441 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.342549 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:39.346839 (+ 4290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:49249: BlockingRecv error: recv got EOF from 127.13.162.1:49249 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 10:02:39 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:39 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:39.539865 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:39.531151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:34759)
0924 10:02:39.531747 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:39.531763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:39.531909 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:39.532229 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:39.532232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:39.532615 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:39.532770 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:39.532778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.533718 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.533722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:39.534771 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:39.534783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.535527 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.535532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:39.535691 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:39.536368 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:39.536398 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:39.539625 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":369,"threads_started":1}
W20260924 10:02:39.540060 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:39.531229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:34759 (local address 127.13.162.62:42009)
0924 10:02:39.531386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:39.531391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:39.531870 (+ 479us) server_negotiation.cc:407] Connection header received
0924 10:02:39.532013 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:39.532017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:39.532114 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:39.532200 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:39.532895 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.533600 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.534880 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.535272 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.535426 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:39.539837 (+ 4411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:34759: BlockingRecv error: recv got EOF from 127.13.162.3:34759 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:02:39 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:39 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:39.975443 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:39.967527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:36491)
0924 10:02:39.967978 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:39.968011 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:39.968122 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:39.968375 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:39.968378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:39.968752 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:39.968894 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:39.968903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.969818 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.969822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:39.970759 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:39.970770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.971353 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.971357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:39.971471 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:39.971984 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:39.972006 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:39.975227 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
W20260924 10:02:39.975610 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:39.967666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:36491 (local address 127.13.162.62:42009)
0924 10:02:39.967807 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:39.967811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:39.968070 (+ 259us) server_negotiation.cc:407] Connection header received
0924 10:02:39.968206 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:39.968209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:39.968266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:39.968348 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:39.969038 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.969721 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.970854 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:39.971253 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:39.971337 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:39.975406 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:36491: BlockingRecv error: recv got EOF from 127.13.162.2:36491 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 10:02:40 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:40 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:40.356829 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:40.347988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:52991)
0924 10:02:40.348424 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:40.348447 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:40.348563 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:40.348944 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:40.348948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:40.349319 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:40.349507 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:40.349518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.350617 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.350621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:40.351516 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:40.351528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.352230 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.352235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:40.352388 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:40.352979 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:40.353000 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:40.356563 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
W20260924 10:02:40.356966 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:40.348042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52991 (local address 127.13.162.62:42009)
0924 10:02:40.348269 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:40.348275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:40.348524 (+ 249us) server_negotiation.cc:407] Connection header received
0924 10:02:40.348758 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:40.348762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:40.348828 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:40.348922 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:40.349685 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.350447 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.351650 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.352084 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.352191 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:40.356733 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:52991: BlockingRecv error: recv got EOF from 127.13.162.1:52991 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 10:02:40 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:40 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:40.549239 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:40.540709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:47557)
0924 10:02:40.541176 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:40.541193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:40.541304 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:40.541594 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:40.541597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:40.542053 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:40.542200 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:40.542209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.543306 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.543311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:40.544324 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:40.544340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.545073 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.545077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:40.545211 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:40.545751 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:40.545769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:40.548962 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
W20260924 10:02:40.549429 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:40.540803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:47557 (local address 127.13.162.62:42009)
0924 10:02:40.540983 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:40.540989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:40.541257 (+ 268us) server_negotiation.cc:407] Connection header received
0924 10:02:40.541396 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:40.541400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:40.541470 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:40.541556 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:40.542367 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.543136 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.544468 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.544943 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.545036 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:40.549212 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:47557: BlockingRecv error: recv got EOF from 127.13.162.3:47557 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:02:40 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:40 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:40.984777 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:40.976293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:56623)
0924 10:02:40.977029 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:40.977044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:40.977179 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:40.977540 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:40.977546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:40.977919 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:40.978086 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:40.978095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.979082 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.979086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:40.979998 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:40.980010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.980680 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.980684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:40.980812 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:40.981350 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:40.981367 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:40.984518 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":513,"threads_started":1}
W20260924 10:02:40.985014 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:40.976381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:56623 (local address 127.13.162.62:42009)
0924 10:02:40.976583 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:40.976590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:40.977124 (+ 534us) server_negotiation.cc:407] Connection header received
0924 10:02:40.977301 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:40.977306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:40.977383 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:40.977477 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:40.978199 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.978959 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.980129 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:40.980545 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:40.980647 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:40.984784 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:56623: BlockingRecv error: recv got EOF from 127.13.162.2:56623 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:02:41 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:41 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:41.366673 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:41.357706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:37787)
0924 10:02:41.358366 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:41.358387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:41.358520 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:41.358840 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:41.358845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:41.359275 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:41.359442 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:41.359451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.360437 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.360443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:41.361383 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:41.361396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.362056 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.362062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:41.362233 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:41.362874 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:41.362892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:41.366341 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":245,"threads_started":1}
W20260924 10:02:41.366776 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:41.357750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:37787 (local address 127.13.162.62:42009)
0924 10:02:41.357923 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:41.357927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:41.358465 (+ 538us) server_negotiation.cc:407] Connection header received
0924 10:02:41.358637 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:41.358642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:41.358717 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:41.358793 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:41.359584 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.360299 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.361517 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.361915 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.362018 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:41.366574 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:37787: BlockingRecv error: recv got EOF from 127.13.162.1:37787 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:02:41 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:41 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:41.559522 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:41.550309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:60113)
0924 10:02:41.550807 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:41.550827 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:41.550953 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:41.551334 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:41.551337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:41.551734 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:41.551902 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:41.551911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.553211 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.553217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:41.554365 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:41.554380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.555146 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.555151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:41.555314 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:41.555898 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:41.555917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:41.559292 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
W20260924 10:02:41.559749 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:41.550507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:60113 (local address 127.13.162.62:42009)
0924 10:02:41.550681 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:41.550686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:41.550909 (+ 223us) server_negotiation.cc:407] Connection header received
0924 10:02:41.551117 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:41.551122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:41.551202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:41.551299 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:41.552051 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.553046 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.554500 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.554924 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.555077 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:41.559505 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:60113: BlockingRecv error: recv got EOF from 127.13.162.3:60113 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:02:41 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:41 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:41.994387 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:41.985643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:46707)
0924 10:02:41.986234 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:41.986249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:41.986388 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:41.986746 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:41.986750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:41.987171 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:41.987337 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:41.987346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.988299 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.988303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:41.989254 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:41.989265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.990028 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.990032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:41.990143 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:41.990667 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:41.990684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:41.994019 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":282,"threads_started":1}
W20260924 10:02:41.994439 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:41.985777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:46707 (local address 127.13.162.62:42009)
0924 10:02:41.985997 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:41.986002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:41.986370 (+ 368us) server_negotiation.cc:407] Connection header received
0924 10:02:41.986512 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:41.986516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:41.986576 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:41.986654 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:41.987472 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.988167 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.989388 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:41.989906 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:41.990002 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:41.994216 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:46707: BlockingRecv error: recv got EOF from 127.13.162.2:46707 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 24 10:02:42 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:42 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:42.375818 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:42.367580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:51139)
0924 10:02:42.368131 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:42.368151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:42.368258 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:42.368527 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:42.368530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:42.368877 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:42.369065 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:42.369074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.370062 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.370066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:42.371028 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:42.371040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.371725 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.371729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:42.371835 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:42.372448 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:42.372465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:42.375582 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":300,"threads_started":1}
W20260924 10:02:42.376071 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:42.367645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51139 (local address 127.13.162.62:42009)
0924 10:02:42.367863 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:42.367869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:42.368227 (+ 358us) server_negotiation.cc:407] Connection header received
0924 10:02:42.368343 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:42.368358 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:42.368420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:42.368501 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:42.369191 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.369929 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.371148 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.371583 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.371676 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:42.375788 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:51139: BlockingRecv error: recv got EOF from 127.13.162.1:51139 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 10:02:42 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:42 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:42.569521 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:42.560418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:50789)
0924 10:02:42.561055 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:42.561071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:42.561183 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:42.561467 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:42.561471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:42.561881 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:42.562064 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:42.562073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.563091 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.563095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:42.564071 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:42.564084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.564768 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.564773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:42.564912 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:42.565510 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:42.565528 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:42.569177 (+ 3649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":330,"threads_started":1}
W20260924 10:02:42.569644 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:42.560611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:50789 (local address 127.13.162.62:42009)
0924 10:02:42.560821 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:42.560825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:42.561172 (+ 347us) server_negotiation.cc:407] Connection header received
0924 10:02:42.561281 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:42.561284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:42.561357 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:42.561430 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:42.562186 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.562952 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.564199 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.564604 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.564700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:42.569403 (+ 4703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:50789: BlockingRecv error: recv got EOF from 127.13.162.3:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 10:02:42 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:42 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:43.004518 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:42.995480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:56277)
0924 10:02:42.996074 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:42.996092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:42.996203 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:42.996537 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:42.996542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:42.996925 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:42.997094 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:42.997102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.998350 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.998355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:42.999496 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:42.999513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.000289 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.000294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:43.000429 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:43.001072 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:43.001089 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:43.004281 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":333,"threads_started":1}
W20260924 10:02:43.004711 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:42.995613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:56277 (local address 127.13.162.62:42009)
0924 10:02:42.995771 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:42.995775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:42.996160 (+ 385us) server_negotiation.cc:407] Connection header received
0924 10:02:42.996300 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:42.996304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:42.996363 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:42.996447 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:42.997247 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:42.998178 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:42.999664 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.000104 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.000201 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:43.004480 (+ 4279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:56277: BlockingRecv error: recv got EOF from 127.13.162.2:56277 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:02:43 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:43 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:43.385133 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:43.376687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:49599)
0924 10:02:43.377160 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:43.377178 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:43.377309 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:43.377638 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:43.377643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:43.378020 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:43.378172 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:43.378181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.379193 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.379197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:43.380141 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:43.380153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.380829 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.380835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:43.380994 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:43.381564 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:43.381582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:43.384824 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":246,"threads_started":1}
W20260924 10:02:43.385351 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:43.376772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:49599 (local address 127.13.162.62:42009)
0924 10:02:43.376969 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:43.376975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:43.377251 (+ 276us) server_negotiation.cc:407] Connection header received
0924 10:02:43.377419 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:43.377423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:43.377503 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:43.377583 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:43.378340 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.379067 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.380270 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.380687 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.380796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:43.385088 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:49599: BlockingRecv error: recv got EOF from 127.13.162.1:49599 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:02:43 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:43 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:43.578925 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:43.570445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:42223)
0924 10:02:43.570945 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:43.570965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:43.571078 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:43.571357 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:43.571361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:43.571721 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:43.571885 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:43.571893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.572974 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.572978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:43.573971 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:43.573983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.574725 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.574729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:43.574841 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:43.575430 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:43.575449 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:43.578681 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
W20260924 10:02:43.579186 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:43.570906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:42223 (local address 127.13.162.62:42009)
0924 10:02:43.571052 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:43.571056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:43.571073 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:02:43.571161 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:43.571165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:43.571233 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:43.571309 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:43.572052 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.572826 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.574113 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:43.574569 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:43.574673 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:43.578899 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:42223: BlockingRecv error: recv got EOF from 127.13.162.3:42223 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 10:02:44 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:44 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:44.017655 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:44.005339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:44243)
0924 10:02:44.005811 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:44.005829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:44.005950 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:44.006256 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:44.006260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:44.006671 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:44.006812 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:44.006822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.007781 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.007785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:44.008710 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:44.008722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.009379 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.009385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:44.009537 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:44.013926 (+ 4389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:44.013950 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:44.017436 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":246,"threads_started":1}
W20260924 10:02:44.017870 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:44.005434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:44243 (local address 127.13.162.62:42009)
0924 10:02:44.005618 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:44.005625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:44.005907 (+ 282us) server_negotiation.cc:407] Connection header received
0924 10:02:44.006060 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:44.006065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:44.006125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:44.006198 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:44.006957 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.007658 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.008846 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.009241 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.009341 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:44.017638 (+ 8297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:44243: BlockingRecv error: recv got EOF from 127.13.162.2:44243 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 10:02:44 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:44 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:44.395049 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:44.386097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:60237)
0924 10:02:44.386609 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:44.386629 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:44.386743 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:44.387044 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:44.387048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:44.387444 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:44.387599 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:44.387608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.388742 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.388746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:44.389832 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:44.389845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.390587 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.390592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:44.390712 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:44.391318 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:44.391340 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:44.394817 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":264,"threads_started":1}
W20260924 10:02:44.395171 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:44.386183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:60237 (local address 127.13.162.62:42009)
0924 10:02:44.386434 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:44.386438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:44.386705 (+ 267us) server_negotiation.cc:407] Connection header received
0924 10:02:44.386844 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:44.386848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:44.386915 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:44.387002 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:44.387770 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.388617 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.389970 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.390451 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.390546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:44.394969 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:60237: BlockingRecv error: recv got EOF from 127.13.162.1:60237 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 10:02:44 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:44 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:44.589013 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:44.579906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:39509)
0924 10:02:44.580433 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:44.580451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:44.580579 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:44.580913 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:44.580917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:44.581313 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:44.581474 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:44.581484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.582571 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.582575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:44.583597 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:44.583612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.584365 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.584371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:44.584499 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:44.585120 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:44.585139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:44.588774 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":276,"threads_started":1}
W20260924 10:02:44.589154 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:44.580002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:39509 (local address 127.13.162.62:42009)
0924 10:02:44.580175 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:44.580180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:44.580577 (+ 397us) server_negotiation.cc:407] Connection header received
0924 10:02:44.580670 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:44.580674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:44.580737 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:44.580823 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:44.581634 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.582405 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.583743 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:44.584166 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:44.584259 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:44.588969 (+ 4710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:39509: BlockingRecv error: recv got EOF from 127.13.162.3:39509 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:02:45 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:45 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:45.027411 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:45.018533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:34453)
0924 10:02:45.019004 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:45.019043 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:45.019167 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:45.019465 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:45.019469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:45.019852 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:45.020000 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:45.020018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.020959 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.020962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:45.021907 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:45.021918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.022676 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.022682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:45.022832 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:45.023376 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:45.023392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:45.027179 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
W20260924 10:02:45.027602 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:45.018618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:34453 (local address 127.13.162.62:42009)
0924 10:02:45.018795 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:45.018801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:45.019123 (+ 322us) server_negotiation.cc:407] Connection header received
0924 10:02:45.019279 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:45.019284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:45.019342 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:45.019419 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:45.020141 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.020823 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.022040 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.022537 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.022648 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:45.027353 (+ 4705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:34453: BlockingRecv error: recv got EOF from 127.13.162.2:34453 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:02:45 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:45 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:45.404690 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:45.396028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:35169)
0924 10:02:45.396488 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:45.396514 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:45.396645 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:45.396943 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:45.396947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:45.397364 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:45.397542 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:45.397552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.398576 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.398583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:45.399497 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:45.399508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.400165 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.400171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:45.400339 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:45.400864 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:45.400880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:45.404478 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":245,"threads_started":1}
W20260924 10:02:45.404860 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:45.396263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:35169 (local address 127.13.162.62:42009)
0924 10:02:45.396490 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:45.396495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:45.396575 (+ 80us) server_negotiation.cc:407] Connection header received
0924 10:02:45.396756 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:45.396762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:45.396825 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:45.396900 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:45.397678 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.398443 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.399620 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.400028 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.400131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:45.404650 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:35169: BlockingRecv error: recv got EOF from 127.13.162.1:35169 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 24 10:02:45 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:45 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:45.598739 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:45.589893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:58715)
0924 10:02:45.590437 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:45.590451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:45.590619 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:45.590927 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:45.590931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:45.591294 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:45.591445 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:45.591453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.592530 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.592534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:45.594138 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:45.594151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.594862 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.594867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:45.594980 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:45.595520 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:45.595539 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:45.598500 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":308,"threads_started":1}
W20260924 10:02:45.598917 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:45.589988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:58715 (local address 127.13.162.62:42009)
0924 10:02:45.590214 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:45.590220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:45.590532 (+ 312us) server_negotiation.cc:407] Connection header received
0924 10:02:45.590738 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:45.590743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:45.590812 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:45.590887 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:45.591584 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.592361 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.594265 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:45.594729 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:45.594823 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:45.598678 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:58715: BlockingRecv error: recv got EOF from 127.13.162.3:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 10:02:46 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:46 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:46.036909 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:46.028279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:36123)
0924 10:02:46.028760 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:46.028775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:46.028903 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:46.029227 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:46.029232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:46.029672 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:46.029860 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:46.029869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.030928 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.030933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:46.031822 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:46.031833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.032498 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.032504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:46.032666 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:46.033229 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:46.033245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:46.036696 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
W20260924 10:02:46.037062 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:46.028409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:36123 (local address 127.13.162.62:42009)
0924 10:02:46.028679 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:46.028685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:46.028847 (+ 162us) server_negotiation.cc:407] Connection header received
0924 10:02:46.029021 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:46.029026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:46.029099 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:46.029191 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:46.029998 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.030811 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.031949 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.032361 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.032477 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:46.036874 (+ 4397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:36123: BlockingRecv error: recv got EOF from 127.13.162.2:36123 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 10:02:46 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:46 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:46.414165 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:46.405578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:53371)
0924 10:02:46.406110 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:46.406130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:46.406253 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:46.406598 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:46.406601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:46.406985 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:46.407184 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:46.407194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.408513 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.408518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:46.409442 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:46.409453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.410139 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.410143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:46.410286 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:46.410883 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:46.410899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:46.413933 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":304,"threads_started":1}
W20260924 10:02:46.414395 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:46.405686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:53371 (local address 127.13.162.62:42009)
0924 10:02:46.405887 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:46.405892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:46.406227 (+ 335us) server_negotiation.cc:407] Connection header received
0924 10:02:46.406404 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:46.406408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:46.406477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:46.406568 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:46.407316 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.408365 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.409577 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.410000 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.410093 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:46.414130 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:53371: BlockingRecv error: recv got EOF from 127.13.162.1:53371 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 24 10:02:46 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:46 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:46.608167 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:46.599648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:32955)
0924 10:02:46.600138 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:46.600154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:46.600292 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:46.600612 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:46.600617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:46.601066 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:46.601258 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:46.601268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.602215 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.602228 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:46.603271 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:46.603284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.603980 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.603986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:46.604159 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:46.604725 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:46.604743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:46.607954 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":294,"threads_started":1}
W20260924 10:02:46.608327 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:46.599777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:32955 (local address 127.13.162.62:42009)
0924 10:02:46.599947 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:46.599954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:46.600276 (+ 322us) server_negotiation.cc:407] Connection header received
0924 10:02:46.600405 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:46.600408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:46.600471 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:46.600559 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:46.601398 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.602094 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.603406 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:46.603847 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:46.603950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:46.608106 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:32955: BlockingRecv error: recv got EOF from 127.13.162.3:32955 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 10:02:47 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:47 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:47.046808 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:47.037888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:53073)
0924 10:02:47.038471 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:47.038487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:47.038598 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:47.038915 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:47.038918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:47.039304 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:47.039471 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:47.039480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.040614 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.040621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:47.041630 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:47.041643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.042485 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.042492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:47.042656 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:47.043197 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:47.043214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:47.046482 (+ 3268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":282,"threads_started":1}
W20260924 10:02:47.046984 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:47.037984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:53073 (local address 127.13.162.62:42009)
0924 10:02:47.038145 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:47.038150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:47.038549 (+ 399us) server_negotiation.cc:407] Connection header received
0924 10:02:47.038709 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:47.038713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:47.038772 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:47.038866 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:47.039610 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.040451 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.041764 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.042273 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.042433 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:47.046684 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:53073: BlockingRecv error: recv got EOF from 127.13.162.2:53073 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:02:47 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:47 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:47.424527 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:47.415183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:43155)
0924 10:02:47.415702 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:47.415724 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:47.415890 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:47.416255 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:47.416258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:47.416637 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:47.416777 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:47.416786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.418201 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.418206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:47.419444 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:47.419460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.420270 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.420274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:47.420413 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:47.421037 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:47.421055 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:47.424287 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
W20260924 10:02:47.424685 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:47.415312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:43155 (local address 127.13.162.62:42009)
0924 10:02:47.415496 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:47.415502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:47.415804 (+ 302us) server_negotiation.cc:407] Connection header received
0924 10:02:47.416014 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:47.416020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:47.416105 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:47.416215 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:47.416966 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.418034 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.419594 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.420140 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.420233 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:47.424469 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:43155: BlockingRecv error: recv got EOF from 127.13.162.1:43155 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 10:02:47 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:47 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:47.618065 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:47.609221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:57123)
0924 10:02:47.609781 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:47.609797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:47.609937 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:47.610288 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:47.610293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:47.610729 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:47.610891 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:47.610899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.611923 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.611928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:47.612848 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:47.612860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.613594 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.613600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:47.613757 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:47.614352 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:47.614370 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:47.617794 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":335,"threads_started":1}
W20260924 10:02:47.618185 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:47.609335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:57123 (local address 127.13.162.62:42009)
0924 10:02:47.609527 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:47.609533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:47.609885 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:02:47.610080 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:47.610085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:47.610163 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:47.610255 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:47.611035 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.611777 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.612994 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:47.613420 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:47.613528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:47.617974 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:57123: BlockingRecv error: recv got EOF from 127.13.162.3:57123 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:02:48 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:48 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:48.056552 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:48.047779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:56475)
0924 10:02:48.048204 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:48.048224 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:48.048347 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:48.048765 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:48.048770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:48.049247 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:48.049417 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:48.049428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.050634 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.050638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:48.051575 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:48.051587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.052302 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.052306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:48.052422 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:48.052972 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:48.052990 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:48.056290 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
W20260924 10:02:48.056727 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:48.047901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:56475 (local address 127.13.162.62:42009)
0924 10:02:48.048114 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:48.048120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:48.048296 (+ 176us) server_negotiation.cc:407] Connection header received
0924 10:02:48.048443 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:48.048446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:48.048527 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:48.048617 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:48.049589 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.050484 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.051708 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.052188 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.052285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:48.056510 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:56475: BlockingRecv error: recv got EOF from 127.13.162.2:56475 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 10:02:48 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:48 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.1@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:48.434428 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:48.425828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.1:52147)
0924 10:02:48.426501 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:48.426520 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:48.426625 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:48.426955 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:48.426958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:48.427315 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:48.427492 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:48.427501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.428550 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.428554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:48.429520 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:48.429533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.430240 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.430244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:48.430386 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:48.430972 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:48.430994 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:48.434114 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":405,"threads_started":1}
W20260924 10:02:48.434544 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:48.425919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52147 (local address 127.13.162.62:42009)
0924 10:02:48.426148 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:48.426155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:48.426596 (+ 441us) server_negotiation.cc:407] Connection header received
0924 10:02:48.426762 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:48.426766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:48.426837 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:48.426923 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:48.427624 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.428438 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.429635 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.430118 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.430210 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:48.434310 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:52147: BlockingRecv error: recv got EOF from 127.13.162.1:52147 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 10:02:48 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:48 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.3@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:48.628448 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:48.619225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.3:36119)
0924 10:02:48.619696 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:48.619713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:48.619842 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:48.620200 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:48.620203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:48.620589 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:48.620764 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:48.620774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.622065 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.622071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:48.623285 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:48.623298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.623979 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.623983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:48.624111 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:48.624694 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:48.624713 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:48.628092 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
W20260924 10:02:48.628540 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:48.619288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:36119 (local address 127.13.162.62:42009)
0924 10:02:48.619573 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:48.619577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:48.619792 (+ 215us) server_negotiation.cc:407] Connection header received
0924 10:02:48.619993 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:48.619998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:48.620073 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:48.620170 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:48.620936 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.621903 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.623429 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:48.623838 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:48.623939 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:48.628335 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.3:36119: BlockingRecv error: recv got EOF from 127.13.162.3:36119 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 24 10:02:49 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:02:49 dist-test-slave-jvsj krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.162.2@KRBTEST.COM for oryx/127.13.162.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:02:49.070467 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:49.057728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:42009 (local address 127.13.162.2:40933)
0924 10:02:49.058181 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:49.058195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:49.058370 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:49.058675 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:49.058679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:49.059026 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:49.059196 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:49.059206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.060147 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.060151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:49.061045 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:49.061055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.061737 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.061743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:49.061906 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:49.066867 (+ 4961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:49.066886 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:49.070054 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.162.62:42009: Server oryx/127.13.162.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
W20260924 10:02:49.070560 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:49.057831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:40933 (local address 127.13.162.62:42009)
0924 10:02:49.058038 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:49.058043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:49.058299 (+ 256us) server_negotiation.cc:407] Connection header received
0924 10:02:49.058486 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:49.058489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:49.058547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:49.058621 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:49.059340 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.060019 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.061169 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.061609 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.061711 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:49.070291 (+ 8580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.2:40933: BlockingRecv error: recv got EOF from 127.13.162.2:40933 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:02:49.424520 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22776
I20260924 10:02:49.436995 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:49.425632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.13.162.1:41843)
0924 10:02:49.426281 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:49.426288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:49.426339 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:02:49.426414 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:49.426427 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:49.426661 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:49.426838 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:49.428169 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.429215 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.430541 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.431066 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.431159 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:49.433467 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:49.433475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:49.433488 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:49.433534 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:49.436286 (+ 2752us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.162.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 10:02:49.436677 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37536: Request ticket server kudu/127.13.162.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":335,"threads_started":1}
W20260924 10:02:49.437022 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22776: Not authorized: Client connection negotiation failed: client connection to 127.13.162.1:41843: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.162.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 10:02:49.437101 23237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37536: Request ticket server kudu/127.13.162.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 10:02:49.437216 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22776
I20260924 10:02:49.451081 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22927
W20260924 10:02:49.451190 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:49.435435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:49519 (local address 127.13.162.62:42009)
0924 10:02:49.435612 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:49.435617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:49.435955 (+ 338us) server_negotiation.cc:407] Connection header received
0924 10:02:49.436070 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:49.436076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:49.436149 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:49.436247 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:49.437147 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.438103 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.450953 (+ 12850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.162.1:49519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:02:49.462997 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:49.452102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.13.162.2:36269)
0924 10:02:49.452704 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:49.452711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:49.452738 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:02:49.452804 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:49.452812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:49.453046 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:49.453246 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:49.454491 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.455561 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.456801 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.457395 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.457506 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:49.459888 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:49.459897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:49.459910 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:49.459951 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:49.462491 (+ 2540us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.162.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 10:02:49.462770 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45010: Request ticket server kudu/127.13.162.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
W20260924 10:02:49.463088 23239 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45010: Request ticket server kudu/127.13.162.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 10:02:49.463086 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22927: Not authorized: Client connection negotiation failed: client connection to 127.13.162.2:36269: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.162.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 10:02:49.463210 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22927
I20260924 10:02:49.476145 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23060
I20260924 10:02:49.488451 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:49.477217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.13.162.3:34607)
0924 10:02:49.477767 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:49.477774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:49.477802 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:02:49.477868 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:49.477874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:49.478105 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:49.478283 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:49.479668 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.480662 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.481865 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:49.482557 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:49.482678 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:49.485111 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:49.485119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:49.485133 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:49.485199 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:49.487778 (+ 2579us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.162.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 10:02:49.488113 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38042: Request ticket server kudu/127.13.162.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":267,"threads_started":1}
W20260924 10:02:49.488564 23240 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38042: Request ticket server kudu/127.13.162.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 10:02:49.488586 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23060: Not authorized: Client connection negotiation failed: client connection to 127.13.162.3:34607: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.162.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 10:02:49.488857 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23060
I20260924 10:02:49.502487 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 22707
I20260924 10:02:49.503760 22769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:49.616910 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 22707
2026-09-24T10:02:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21053 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T10:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:49Z Disabled control of system clock
I20260924 10:02:49.685166 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36697
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:02:50.000564 23245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:50.000864 23245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:50.000965 23245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:50.008378 23245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:50.008500 23245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:50.008541 23245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:50.008620 23245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:50.008677 23245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:50.018090 23245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:50.019443 23245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:50.021247 23245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:50.033291 23250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:50.033300 23253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:50.033381 23251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:50.033830 23245 server_base.cc:1034] running on GCE node
I20260924 10:02:50.034788 23245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:50.036183 23245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:50.037384 23245 hybrid_clock.cc:648] HybridClock initialized: now 1790244170037328 us; error 52 us; skew 500 ppm
I20260924 10:02:50.037717 23245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:50.039907 23245 webserver.cc:460] Webserver started at http://127.13.162.62:42085/ using document root <none> and password file <none>
I20260924 10:02:50.040578 23245 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:50.040675 23245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:50.040977 23245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:50.043800 23245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "82003d3192d843bcabd73b769d541ce0"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "20aa6ea9cc7c3f1e94eb07cfb924a9f7"
server_key_iv: "3f7939c93058f3fad15b8fbf1a00e555"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.044533 23245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "82003d3192d843bcabd73b769d541ce0"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "20aa6ea9cc7c3f1e94eb07cfb924a9f7"
server_key_iv: "3f7939c93058f3fad15b8fbf1a00e555"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.049304 23245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:02:50.052472 23259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:50.053687 23245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:02:50.053838 23245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "82003d3192d843bcabd73b769d541ce0"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "20aa6ea9cc7c3f1e94eb07cfb924a9f7"
server_key_iv: "3f7939c93058f3fad15b8fbf1a00e555"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.054019 23245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:50.069727 23245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:50.070703 23245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:50.070971 23245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:50.087325 23245 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:36697
I20260924 10:02:50.087332 23310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:36697 every 8 connection(s)
I20260924 10:02:50.088713 23245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:50.092310 23311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:50.094722 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23245
I20260924 10:02:50.094893 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:50.095670 13960 external_mini_cluster.cc:1428] Setting key 0a804483e6561534bec12de5930e83dd
I20260924 10:02:50.105446 23311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0: Bootstrap starting.
I20260924 10:02:50.109171 23311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:50.110450 23311 log.cc:824] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:50.110913 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:50.097946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.13.162.62:36697)
0924 10:02:50.099042 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:50.099058 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:50.099139 (+ 81us) server_negotiation.cc:407] Connection header received
0924 10:02:50.101109 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:50.101137 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:50.101764 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:50.102195 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:50.102764 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:50.104631 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:50.105946 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:50.106920 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:50.107120 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:50.108131 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:50.108140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:50.108158 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:50.108828 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:50.108942 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:50.108968 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:50.109089 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:50.109305 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 10:02:50.109654 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":358,"threads_started":1}
I20260924 10:02:50.113348 23311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0: No bootstrap required, opened a new log
I20260924 10:02:50.120445 23311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82003d3192d843bcabd73b769d541ce0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36697 } }
I20260924 10:02:50.120786 23311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:50.120857 23311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82003d3192d843bcabd73b769d541ce0, State: Initialized, Role: FOLLOWER
I20260924 10:02:50.121376 23311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82003d3192d843bcabd73b769d541ce0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36697 } }
I20260924 10:02:50.121579 23311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:50.121670 23311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:50.121778 23311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:50.123921 23311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82003d3192d843bcabd73b769d541ce0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36697 } }
I20260924 10:02:50.124357 23311 leader_election.cc:302] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82003d3192d843bcabd73b769d541ce0; no voters:
I20260924 10:02:50.124850 23311 leader_election.cc:288] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:50.124994 23316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:50.125630 23316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 82003d3192d843bcabd73b769d541ce0, State: Running, Role: LEADER
I20260924 10:02:50.126132 23316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82003d3192d843bcabd73b769d541ce0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36697 } }
I20260924 10:02:50.126762 23311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:50.128705 23318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82003d3192d843bcabd73b769d541ce0. Latest consensus state: current_term: 1 leader_uuid: "82003d3192d843bcabd73b769d541ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82003d3192d843bcabd73b769d541ce0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36697 } } }
I20260924 10:02:50.129084 23318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:50.129604 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82003d3192d843bcabd73b769d541ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82003d3192d843bcabd73b769d541ce0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36697 } } }
I20260924 10:02:50.129959 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:50.131155 23325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:50.136159 23325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:50.144727 23325 catalog_manager.cc:1347] Generated new cluster ID: a49f42210b4f4d2f8379bf9dd780eb8c
I20260924 10:02:50.144856 23325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:50.179736 23325 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:50.181717 23325 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:50.189531 23325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82003d3192d843bcabd73b769d541ce0: Generated new TSK 0
I20260924 10:02:50.190541 23325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:02:50.196434 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36697
--builtin_ntp_servers=127.13.162.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 10:02:50.517868 23335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:50.518146 23335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:50.518268 23335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:50.525678 23335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:50.525792 23335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:50.525924 23335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:50.535521 23335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:50.536823 23335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:50.538640 23335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:50.551415 23340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:50.551491 23341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:50.551458 23343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:50.552721 23335 server_base.cc:1034] running on GCE node
I20260924 10:02:50.553284 23335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:50.554229 23335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:50.555433 23335 hybrid_clock.cc:648] HybridClock initialized: now 1790244170555364 us; error 63 us; skew 500 ppm
I20260924 10:02:50.555768 23335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:50.558032 23335 webserver.cc:460] Webserver started at http://127.13.162.1:44939/ using document root <none> and password file <none>
I20260924 10:02:50.558768 23335 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:50.558866 23335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:50.559157 23335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:50.561913 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "bd1fbf43b8664af5bb667002a4354c0a"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "131a7d462d36149864a1033b20aadcdb"
server_key_iv: "04130f6f9f72be3dcdcb1cfe2229084e"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.562713 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "bd1fbf43b8664af5bb667002a4354c0a"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "131a7d462d36149864a1033b20aadcdb"
server_key_iv: "04130f6f9f72be3dcdcb1cfe2229084e"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.567934 23335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:02:50.571429 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:50.572774 23335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:50.572928 23335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "bd1fbf43b8664af5bb667002a4354c0a"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "131a7d462d36149864a1033b20aadcdb"
server_key_iv: "04130f6f9f72be3dcdcb1cfe2229084e"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.573159 23335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:50.590698 23335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:50.591519 23335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:50.591789 23335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:50.592703 23335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:50.594094 23335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:50.594187 23335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:50.594292 23335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:50.594378 23335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 10:02:50.624155 23335 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:36365
I20260924 10:02:50.624207 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:36365 every 8 connection(s)
I20260924 10:02:50.625437 23335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:50.629041 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23335
I20260924 10:02:50.629206 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:50.629841 13960 external_mini_cluster.cc:1428] Setting key 3930576c071c3eb24e8b29110a80f6f1
I20260924 10:02:50.632817 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36697
--builtin_ntp_servers=127.13.162.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:02:50.639248 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:50.627969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:53815 (local address 127.13.162.62:36697)
0924 10:02:50.628350 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:50.628355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:50.629899 (+ 1544us) server_negotiation.cc:407] Connection header received
0924 10:02:50.632224 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:50.632229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:50.632309 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:50.632428 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:50.633579 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:50.634602 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:50.636565 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:50.637170 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:50.637283 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:50.638555 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:50.638561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:50.638565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:50.638588 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:50.638634 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:50.638649 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:50.638737 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:50.638945 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 10:02:50.639019 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":189,"threads_started":1}
I20260924 10:02:50.640581 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:50.628667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36697 (local address 127.13.162.1:53815)
0924 10:02:50.629725 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:50.629776 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:50.631945 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:50.632652 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:50.632661 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:50.632754 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:50.633395 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:50.633413 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:50.634740 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:50.634746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:50.636409 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:50.636424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:50.637352 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:50.637357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:50.637599 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:50.638239 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:50.638241 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:50.638281 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:50.638390 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:50.638732 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:50.638738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:50.638743 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:02:50.638929 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:02:50.639334 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":298,"threads_started":1}
I20260924 10:02:50.641950 23462 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36697
I20260924 10:02:50.642251 23462 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:50.643167 23462 heartbeater.cc:511] Master 127.13.162.62:36697 requested a full tablet report, sending...
I20260924 10:02:50.645244 23276 ts_manager.cc:194] Registered new tserver with Master: bd1fbf43b8664af5bb667002a4354c0a (127.13.162.1:36365)
I20260924 10:02:50.646762 23276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.1:53815
W20260924 10:02:50.943329 23466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:50.943627 23466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:50.943749 23466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:50.951195 23466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:50.951300 23466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:50.951437 23466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:50.960518 23466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:50.961747 23466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:50.963449 23466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:50.975497 23471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:50.975535 23472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:50.975571 23474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:50.976315 23466 server_base.cc:1034] running on GCE node
I20260924 10:02:50.976893 23466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:50.977875 23466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:50.979075 23466 hybrid_clock.cc:648] HybridClock initialized: now 1790244170979033 us; error 58 us; skew 500 ppm
I20260924 10:02:50.979374 23466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:50.981626 23466 webserver.cc:460] Webserver started at http://127.13.162.2:43965/ using document root <none> and password file <none>
I20260924 10:02:50.982295 23466 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:50.982424 23466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:50.982739 23466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:50.985525 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "86ac1d3bc5fc4794a7954a60432a10b3"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "316f1bf64fd9fd35b3cc0bb2ea62610d"
server_key_iv: "2d7f625a8fd8606ded49b6be845d62c4"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.986259 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "86ac1d3bc5fc4794a7954a60432a10b3"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "316f1bf64fd9fd35b3cc0bb2ea62610d"
server_key_iv: "2d7f625a8fd8606ded49b6be845d62c4"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.991057 23466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:02:50.994282 23480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:50.995651 23466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:50.995810 23466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "86ac1d3bc5fc4794a7954a60432a10b3"
format_stamp: "Formatted at 2026-09-24 10:02:50 on dist-test-slave-jvsj"
server_key: "316f1bf64fd9fd35b3cc0bb2ea62610d"
server_key_iv: "2d7f625a8fd8606ded49b6be845d62c4"
server_key_version: "encryptionkey@0"
I20260924 10:02:50.996001 23466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:51.015110 23466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:51.015967 23466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:51.016249 23466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:51.017310 23466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:51.018751 23466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:51.018860 23466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:51.018961 23466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:51.019032 23466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:51.048352 23466 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:45371
I20260924 10:02:51.048401 23592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:45371 every 8 connection(s)
I20260924 10:02:51.049621 23466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:51.054725 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23466
I20260924 10:02:51.054872 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:51.055275 13960 external_mini_cluster.cc:1428] Setting key 1b4531dc65f3d71f99e62198c0484b27
I20260924 10:02:51.058300 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36697
--builtin_ntp_servers=127.13.162.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:02:51.062402 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.052059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:56935 (local address 127.13.162.62:36697)
0924 10:02:51.052264 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:51.052270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:51.053647 (+ 1377us) server_negotiation.cc:407] Connection header received
0924 10:02:51.055550 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:51.055556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:51.055638 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:51.055752 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:51.057077 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.058065 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.059790 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.060302 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.060409 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.061661 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:51.061667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:51.061671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:51.061692 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:51.061726 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:51.061739 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:51.061818 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:51.062004 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 10:02:51.062092 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:02:51.063408 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.052611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36697 (local address 127.13.162.2:56935)
0924 10:02:51.053501 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:51.053542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:51.055196 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:51.056009 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:51.056023 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:51.056133 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:51.056889 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:51.056916 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.058223 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.058227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:51.059645 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:51.059658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.060520 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.060526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:51.060731 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.061361 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:51.061363 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:51.061404 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:51.061496 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:51.061828 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:51.061836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:51.061842 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:02:51.061986 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:02:51.062299 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":245,"threads_started":1}
I20260924 10:02:51.064858 23593 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36697
I20260924 10:02:51.065140 23593 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:51.065861 23593 heartbeater.cc:511] Master 127.13.162.62:36697 requested a full tablet report, sending...
I20260924 10:02:51.067219 23276 ts_manager.cc:194] Registered new tserver with Master: 86ac1d3bc5fc4794a7954a60432a10b3 (127.13.162.2:45371)
I20260924 10:02:51.067965 23276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.2:56935
W20260924 10:02:51.379563 23596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:51.379841 23596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:51.379946 23596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:51.387254 23596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:51.387364 23596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:51.387506 23596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:51.396514 23596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:36697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:51.397825 23596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:51.399654 23596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:51.412520 23601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:51.412482 23602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:51.412495 23604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:51.413628 23596 server_base.cc:1034] running on GCE node
I20260924 10:02:51.414284 23596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:51.415277 23596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:51.416497 23596 hybrid_clock.cc:648] HybridClock initialized: now 1790244171416440 us; error 47 us; skew 500 ppm
I20260924 10:02:51.416841 23596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:51.419224 23596 webserver.cc:460] Webserver started at http://127.13.162.3:43029/ using document root <none> and password file <none>
I20260924 10:02:51.419922 23596 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:51.420025 23596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:51.420332 23596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:51.423233 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "b6f9472a87f241dea574110fe9b859e2"
format_stamp: "Formatted at 2026-09-24 10:02:51 on dist-test-slave-jvsj"
server_key: "5647b8e3571fa718b89ac7383e2692f7"
server_key_iv: "83b0de56e44865e40d6c41390138fbf5"
server_key_version: "encryptionkey@0"
I20260924 10:02:51.424028 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "b6f9472a87f241dea574110fe9b859e2"
format_stamp: "Formatted at 2026-09-24 10:02:51 on dist-test-slave-jvsj"
server_key: "5647b8e3571fa718b89ac7383e2692f7"
server_key_iv: "83b0de56e44865e40d6c41390138fbf5"
server_key_version: "encryptionkey@0"
I20260924 10:02:51.429503 23596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:51.433145 23610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:51.434762 23596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:02:51.434930 23596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "b6f9472a87f241dea574110fe9b859e2"
format_stamp: "Formatted at 2026-09-24 10:02:51 on dist-test-slave-jvsj"
server_key: "5647b8e3571fa718b89ac7383e2692f7"
server_key_iv: "83b0de56e44865e40d6c41390138fbf5"
server_key_version: "encryptionkey@0"
I20260924 10:02:51.435158 23596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:51.447314 23596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:51.448135 23596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:51.448407 23596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:51.449319 23596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:51.450747 23596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:51.450840 23596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:51.450943 23596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:51.451009 23596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:51.480283 23596 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:42811
I20260924 10:02:51.480324 23722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:42811 every 8 connection(s)
I20260924 10:02:51.481530 23596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:51.489614 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23596
I20260924 10:02:51.489769 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:51.490187 13960 external_mini_cluster.cc:1428] Setting key 7c6d92c97d358d3292b0ed12140cb8dd
I20260924 10:02:51.493752 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.483948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:47077 (local address 127.13.162.62:36697)
0924 10:02:51.484142 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:51.484148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:51.485573 (+ 1425us) server_negotiation.cc:407] Connection header received
0924 10:02:51.487316 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:51.487322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:51.487391 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:51.487487 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:51.488589 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.489334 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.491125 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.491596 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.491705 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.492989 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:51.492994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:51.492997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:51.493014 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:51.493044 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:51.493055 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:51.493125 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:51.493326 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 10:02:51.493418 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:02:51.494778 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.484524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36697 (local address 127.13.162.3:47077)
0924 10:02:51.485403 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:51.485439 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:51.487041 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:51.487689 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:51.487698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:51.487796 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:51.488384 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:51.488402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.489502 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.489508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:51.490981 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:51.490993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.491807 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.491815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:51.492054 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.492694 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:51.492696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:51.492741 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:51.492819 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:51.493137 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:51.493148 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:51.493156 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:02:51.493314 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 10:02:51.493710 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":243,"threads_started":1}
I20260924 10:02:51.496030 23723 heartbeater.cc:347] Connected to a master server at 127.13.162.62:36697
I20260924 10:02:51.496309 23723 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:51.497016 23723 heartbeater.cc:511] Master 127.13.162.62:36697 requested a full tablet report, sending...
I20260924 10:02:51.498267 23276 ts_manager.cc:194] Registered new tserver with Master: b6f9472a87f241dea574110fe9b859e2 (127.13.162.3:42811)
I20260924 10:02:51.499132 23276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.3:47077
I20260924 10:02:51.504837 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:02:51.511935 23464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:51.510380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.13.162.62:36697)
0924 10:02:51.510537 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:51.510542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:51.510996 (+ 454us) server_negotiation.cc:407] Connection header received
0924 10:02:51.511161 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:51.511167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:51.511243 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:51.511321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:51.511711 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56258: BlockingRecv error: recv got EOF from 127.0.0.1:56258 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:02:51.512710 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23335
I20260924 10:02:51.520010 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.513503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.13.162.1:36365)
0924 10:02:51.513973 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:51.513980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:51.514005 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:02:51.514060 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:51.514067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:51.514277 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:51.514449 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:51.514784 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.515723 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.517055 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.517606 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.517699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.518707 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:51.518714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:51.518727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:51.519296 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:51.519341 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:51.519365 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:51.519446 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:51.519555 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 10:02:51.519676 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":205,"threads_started":1}
I20260924 10:02:51.521222 23457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:51.600343 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23335
I20260924 10:02:51.613509 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23466
I20260924 10:02:51.621546 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.614504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.13.162.2:45371)
0924 10:02:51.614991 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:51.614998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:51.615024 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:02:51.615081 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:51.615087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:51.615307 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:51.615429 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:51.615818 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.617112 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.618537 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.619084 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.619178 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.620162 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:51.620170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:51.620184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:51.620823 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:51.620875 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:51.620894 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:51.620976 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:51.621134 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:02:51.621262 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260924 10:02:51.622822 23588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:51.702107 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23466
I20260924 10:02:51.715210 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23596
I20260924 10:02:51.723109 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:51.716209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.13.162.3:42811)
0924 10:02:51.716810 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:51.716824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:51.716853 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:02:51.716910 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:51.716917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:51.717127 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:51.717275 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:51.717699 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.718840 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.720005 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:51.720544 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:51.720652 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:51.721640 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:51.721649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:51.721663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:51.722228 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:51.722277 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:51.722297 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:51.722497 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:51.722631 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 10:02:51.722758 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
I20260924 10:02:51.724329 23718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:51.807891 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23596
I20260924 10:02:51.820672 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23245
I20260924 10:02:51.821832 23306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:51.919211 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23245
2026-09-24T10:02:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2297 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T10:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:51Z Disabled control of system clock
I20260924 10:02:51.981572 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:46737
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:02:52.289492 23742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:52.289794 23742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:52.289917 23742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:52.297380 23742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:52.297499 23742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:52.297545 23742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:52.297624 23742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:52.297684 23742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:52.307015 23742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:46737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:52.308315 23742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:52.310099 23742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:52.321123 23747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:52.321086 23748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:52.321609 23750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:52.322081 23742 server_base.cc:1034] running on GCE node
I20260924 10:02:52.322913 23742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:52.324307 23742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:52.325520 23742 hybrid_clock.cc:648] HybridClock initialized: now 1790244172325461 us; error 50 us; skew 500 ppm
I20260924 10:02:52.325877 23742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:52.328210 23742 webserver.cc:460] Webserver started at http://127.13.162.62:45815/ using document root <none> and password file <none>
I20260924 10:02:52.328895 23742 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:52.328994 23742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:52.329304 23742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:52.332237 23742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "f2895ec7db24470d99ca64bf76fc282d"
format_stamp: "Formatted at 2026-09-24 10:02:52 on dist-test-slave-jvsj"
server_key: "34103a6a722aec6ffddb60ca3892b791"
server_key_iv: "f90586a14d99387c74d9e9400418d509"
server_key_version: "encryptionkey@0"
I20260924 10:02:52.333073 23742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "f2895ec7db24470d99ca64bf76fc282d"
format_stamp: "Formatted at 2026-09-24 10:02:52 on dist-test-slave-jvsj"
server_key: "34103a6a722aec6ffddb60ca3892b791"
server_key_iv: "f90586a14d99387c74d9e9400418d509"
server_key_version: "encryptionkey@0"
I20260924 10:02:52.338253 23742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:02:52.341784 23756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:52.343149 23742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:52.343323 23742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "f2895ec7db24470d99ca64bf76fc282d"
format_stamp: "Formatted at 2026-09-24 10:02:52 on dist-test-slave-jvsj"
server_key: "34103a6a722aec6ffddb60ca3892b791"
server_key_iv: "f90586a14d99387c74d9e9400418d509"
server_key_version: "encryptionkey@0"
I20260924 10:02:52.343524 23742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:52.348969 23742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:52.349692 23742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:52.349953 23742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:52.366050 23742 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:46737
I20260924 10:02:52.366066 23807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:46737 every 8 connection(s)
I20260924 10:02:52.367327 23742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:52.369563 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23742
I20260924 10:02:52.369752 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:52.370365 13960 external_mini_cluster.cc:1428] Setting key 1e3a10405800c645d7f14ae012b89dbb
I20260924 10:02:52.371050 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:52.378795 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:52.372284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.13.162.62:46737)
0924 10:02:52.373187 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:52.373198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:52.373236 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:02:52.375415 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:52.375436 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:52.375841 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:52.376186 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:52.377211 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:52.377217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:52.377230 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:52.377244 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:52.377311 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:52.377327 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:52.377552 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:52.377671 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 10:02:52.377897 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260924 10:02:52.383706 23808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d: Bootstrap starting.
I20260924 10:02:52.386675 23808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:52.387850 23808 log.cc:824] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:52.390825 23808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d: No bootstrap required, opened a new log
I20260924 10:02:52.398254 23808 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2895ec7db24470d99ca64bf76fc282d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46737 } }
I20260924 10:02:52.398742 23808 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:52.398866 23808 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2895ec7db24470d99ca64bf76fc282d, State: Initialized, Role: FOLLOWER
I20260924 10:02:52.399487 23808 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2895ec7db24470d99ca64bf76fc282d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46737 } }
I20260924 10:02:52.399703 23808 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:52.399803 23808 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:52.399917 23808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:52.402000 23808 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2895ec7db24470d99ca64bf76fc282d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46737 } }
I20260924 10:02:52.402520 23808 leader_election.cc:302] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2895ec7db24470d99ca64bf76fc282d; no voters:
I20260924 10:02:52.403076 23808 leader_election.cc:288] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:52.403215 23813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:52.403865 23813 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [term 1 LEADER]: Becoming Leader. State: Replica: f2895ec7db24470d99ca64bf76fc282d, State: Running, Role: LEADER
I20260924 10:02:52.404423 23813 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2895ec7db24470d99ca64bf76fc282d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46737 } }
I20260924 10:02:52.405066 23808 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:52.407084 23814 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2895ec7db24470d99ca64bf76fc282d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2895ec7db24470d99ca64bf76fc282d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46737 } } }
I20260924 10:02:52.407460 23814 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:52.408257 23821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:52.408151 23815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2895ec7db24470d99ca64bf76fc282d. Latest consensus state: current_term: 1 leader_uuid: "f2895ec7db24470d99ca64bf76fc282d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2895ec7db24470d99ca64bf76fc282d" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46737 } } }
I20260924 10:02:52.409161 23815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:52.413194 23821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:52.421348 23821 catalog_manager.cc:1347] Generated new cluster ID: 59a29f1ba7fe4929b6eba5b2512af1b5
I20260924 10:02:52.421456 23821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:52.449736 23821 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:52.450146 23821 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:52.462512 23821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2895ec7db24470d99ca64bf76fc282d: Generated new TSK 0
I20260924 10:02:52.463603 23821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:02:52.474684 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:46737
--builtin_ntp_servers=127.13.162.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:02:52.786608 23832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:52.786860 23832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:52.786944 23832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:52.793879 23832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:52.793975 23832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:52.794081 23832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:52.802971 23832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:46737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:52.804174 23832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:52.805989 23832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:52.818416 23840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:52.818418 23838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:52.818519 23837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:52.819195 23832 server_base.cc:1034] running on GCE node
I20260924 10:02:52.819758 23832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:52.820724 23832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:52.821938 23832 hybrid_clock.cc:648] HybridClock initialized: now 1790244172821889 us; error 38 us; skew 500 ppm
I20260924 10:02:52.822234 23832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:52.824553 23832 webserver.cc:460] Webserver started at http://127.13.162.1:41139/ using document root <none> and password file <none>
I20260924 10:02:52.825238 23832 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:52.825325 23832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:52.825634 23832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:52.829762 23832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "c5adab1312f84b118d354e0fc9a639a1"
format_stamp: "Formatted at 2026-09-24 10:02:52 on dist-test-slave-jvsj"
server_key: "ace3ac541422ccf3dde7299841d979ab"
server_key_iv: "9dca110fbf05c0bb199412c22a3f7156"
server_key_version: "encryptionkey@0"
I20260924 10:02:52.830816 23832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "c5adab1312f84b118d354e0fc9a639a1"
format_stamp: "Formatted at 2026-09-24 10:02:52 on dist-test-slave-jvsj"
server_key: "ace3ac541422ccf3dde7299841d979ab"
server_key_iv: "9dca110fbf05c0bb199412c22a3f7156"
server_key_version: "encryptionkey@0"
I20260924 10:02:52.835997 23832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:02:52.839555 23846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:52.841023 23832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:02:52.841174 23832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "c5adab1312f84b118d354e0fc9a639a1"
format_stamp: "Formatted at 2026-09-24 10:02:52 on dist-test-slave-jvsj"
server_key: "ace3ac541422ccf3dde7299841d979ab"
server_key_iv: "9dca110fbf05c0bb199412c22a3f7156"
server_key_version: "encryptionkey@0"
I20260924 10:02:52.841387 23832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:52.847259 23832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:52.848080 23832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:52.848300 23832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:52.849128 23832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:52.850497 23832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:52.850579 23832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:52.850656 23832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:52.850703 23832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:52.880169 23832 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:38179
I20260924 10:02:52.880259 23958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:38179 every 8 connection(s)
I20260924 10:02:52.881434 23832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:52.884660 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23832
I20260924 10:02:52.884877 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:52.885643 13960 external_mini_cluster.cc:1428] Setting key 86c9867e3e08e6d9f7cd03b26bf35381
I20260924 10:02:52.888468 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:46737
--builtin_ntp_servers=127.13.162.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:02:52.889529 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:52.883822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51227 (local address 127.13.162.62:46737)
0924 10:02:52.884202 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:52.884207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:52.885550 (+ 1343us) server_negotiation.cc:407] Connection header received
0924 10:02:52.887301 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:52.887306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:52.887365 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:52.887472 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:52.888876 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:52.888880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:52.888883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:52.888899 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:52.888939 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:52.888950 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:52.889039 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:52.889259 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:02:52.889327 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":199,"threads_started":1}
I20260924 10:02:52.890720 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:52.884416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:46737 (local address 127.13.162.1:51227)
0924 10:02:52.885401 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:52.885435 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:52.887015 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:52.887740 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:52.887751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:52.887886 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:52.888560 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:52.888563 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:52.888615 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:52.888725 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:52.889030 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:52.889039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:52.889047 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:02:52.889323 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 10:02:52.889577 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":322,"threads_started":1}
I20260924 10:02:52.892110 23959 heartbeater.cc:347] Connected to a master server at 127.13.162.62:46737
I20260924 10:02:52.892391 23959 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:52.892952 23959 heartbeater.cc:511] Master 127.13.162.62:46737 requested a full tablet report, sending...
I20260924 10:02:52.894827 23773 ts_manager.cc:194] Registered new tserver with Master: c5adab1312f84b118d354e0fc9a639a1 (127.13.162.1:38179)
W20260924 10:02:53.196193 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:53.196480 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:53.196609 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:53.203697 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:53.203817 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:53.203974 23963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:53.213187 23963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:46737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:53.214476 23963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:53.216156 23963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:53.228524 23968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:53.228605 23969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:53.228554 23971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:53.229357 23963 server_base.cc:1034] running on GCE node
I20260924 10:02:53.229918 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:53.230824 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:53.232024 23963 hybrid_clock.cc:648] HybridClock initialized: now 1790244173231978 us; error 42 us; skew 500 ppm
I20260924 10:02:53.232317 23963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:53.234442 23963 webserver.cc:460] Webserver started at http://127.13.162.2:45125/ using document root <none> and password file <none>
I20260924 10:02:53.235103 23963 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:53.235201 23963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:53.235492 23963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:53.238307 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "ebbea335346942368816a9937fe96cbc"
format_stamp: "Formatted at 2026-09-24 10:02:53 on dist-test-slave-jvsj"
server_key: "b66cda725fefa2950d7e41c7c5c3998b"
server_key_iv: "b7bce552da18e66626dd5f3595c4b170"
server_key_version: "encryptionkey@0"
I20260924 10:02:53.239121 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "ebbea335346942368816a9937fe96cbc"
format_stamp: "Formatted at 2026-09-24 10:02:53 on dist-test-slave-jvsj"
server_key: "b66cda725fefa2950d7e41c7c5c3998b"
server_key_iv: "b7bce552da18e66626dd5f3595c4b170"
server_key_version: "encryptionkey@0"
I20260924 10:02:53.244001 23963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:02:53.247351 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:53.248636 23963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:02:53.248780 23963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "ebbea335346942368816a9937fe96cbc"
format_stamp: "Formatted at 2026-09-24 10:02:53 on dist-test-slave-jvsj"
server_key: "b66cda725fefa2950d7e41c7c5c3998b"
server_key_iv: "b7bce552da18e66626dd5f3595c4b170"
server_key_version: "encryptionkey@0"
I20260924 10:02:53.248979 23963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:53.254591 23963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:53.255398 23963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:53.255640 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:53.256523 23963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:53.257916 23963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:53.258006 23963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:53.258107 23963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:53.258173 23963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:53.287634 23963 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:42787
I20260924 10:02:53.287679 24089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:42787 every 8 connection(s)
I20260924 10:02:53.288905 23963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:53.297029 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.291518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:59301 (local address 127.13.162.62:46737)
0924 10:02:53.291687 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:53.291691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:53.293076 (+ 1385us) server_negotiation.cc:407] Connection header received
0924 10:02:53.294717 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:53.294722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:53.294775 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:53.294863 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:53.296331 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:53.296336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:53.296339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:53.296358 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:53.296395 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:53.296407 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:53.296512 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:53.296759 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 10:02:53.296816 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:02:53.298055 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.292007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:46737 (local address 127.13.162.2:59301)
0924 10:02:53.292918 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:53.292960 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:53.294460 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:53.295087 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:53.295095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:53.295189 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:53.295970 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:53.295973 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:53.296014 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:53.296107 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:53.296526 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:53.296536 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:53.296543 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:02:53.296804 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 10:02:53.297017 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":282,"threads_started":1}
I20260924 10:02:53.298727 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 23963
I20260924 10:02:53.298907 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:53.299465 13960 external_mini_cluster.cc:1428] Setting key 9c46f05875c588bf27546bedefe9b3a1
I20260924 10:02:53.299615 24090 heartbeater.cc:347] Connected to a master server at 127.13.162.62:46737
I20260924 10:02:53.299917 24090 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:53.300565 24090 heartbeater.cc:511] Master 127.13.162.62:46737 requested a full tablet report, sending...
I20260924 10:02:53.302057 23773 ts_manager.cc:194] Registered new tserver with Master: ebbea335346942368816a9937fe96cbc (127.13.162.2:42787)
I20260924 10:02:53.302201 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:46737
--builtin_ntp_servers=127.13.162.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:02:53.616168 24093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:53.616453 24093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:53.616570 24093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:53.623975 24093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:53.624090 24093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:53.624230 24093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:53.633277 24093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:46737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:53.634648 24093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:53.636380 24093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:53.648633 24101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:53.648754 24098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:53.648880 24099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:53.649652 24093 server_base.cc:1034] running on GCE node
I20260924 10:02:53.650246 24093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:53.651212 24093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:53.652420 24093 hybrid_clock.cc:648] HybridClock initialized: now 1790244173652359 us; error 51 us; skew 500 ppm
I20260924 10:02:53.652730 24093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:02:53.654912 24093 webserver.cc:460] Webserver started at http://127.13.162.3:33925/ using document root <none> and password file <none>
I20260924 10:02:53.655596 24093 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:53.655694 24093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:53.655992 24093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:53.658842 24093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "839eb6300b2c465da3d1ee39a6acd2fe"
format_stamp: "Formatted at 2026-09-24 10:02:53 on dist-test-slave-jvsj"
server_key: "4c0742d5a5bd3ae962e48cacb20dee75"
server_key_iv: "82bc3c68dfa784fdcc3ed66d673b4ab7"
server_key_version: "encryptionkey@0"
I20260924 10:02:53.659608 24093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "839eb6300b2c465da3d1ee39a6acd2fe"
format_stamp: "Formatted at 2026-09-24 10:02:53 on dist-test-slave-jvsj"
server_key: "4c0742d5a5bd3ae962e48cacb20dee75"
server_key_iv: "82bc3c68dfa784fdcc3ed66d673b4ab7"
server_key_version: "encryptionkey@0"
I20260924 10:02:53.664584 24093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:02:53.667934 24107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:53.669231 24093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:02:53.669377 24093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "839eb6300b2c465da3d1ee39a6acd2fe"
format_stamp: "Formatted at 2026-09-24 10:02:53 on dist-test-slave-jvsj"
server_key: "4c0742d5a5bd3ae962e48cacb20dee75"
server_key_iv: "82bc3c68dfa784fdcc3ed66d673b4ab7"
server_key_version: "encryptionkey@0"
I20260924 10:02:53.669570 24093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:53.675118 24093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:53.675850 24093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:53.676093 24093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:53.676908 24093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:53.678202 24093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:53.678299 24093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:53.678426 24093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:53.678496 24093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:53.707818 24093 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:39897
I20260924 10:02:53.707861 24219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:39897 every 8 connection(s)
I20260924 10:02:53.709180 24093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:53.711891 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 24093
I20260924 10:02:53.712082 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:53.712493 13960 external_mini_cluster.cc:1428] Setting key 662d68ff8f9710c348cea6869827c45f
I20260924 10:02:53.717514 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.711860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:44417 (local address 127.13.162.62:46737)
0924 10:02:53.712093 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:53.712099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:53.713747 (+ 1648us) server_negotiation.cc:407] Connection header received
0924 10:02:53.715455 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:53.715459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:53.715509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:53.715586 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:53.716928 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:53.716931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:53.716934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:53.716947 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:53.716977 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:53.716987 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:53.717076 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:53.717261 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 10:02:53.717304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 10:02:53.718617 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.712505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:46737 (local address 127.13.162.3:44417)
0924 10:02:53.713588 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:53.713632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:53.715217 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:53.715892 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:53.715901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:53.716003 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:53.716656 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:02:53.716658 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:53.716702 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:02:53.716791 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:53.717090 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:53.717099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:53.717105 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:02:53.717327 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 10:02:53.717531 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":315,"threads_started":1}
I20260924 10:02:53.719795 24220 heartbeater.cc:347] Connected to a master server at 127.13.162.62:46737
I20260924 10:02:53.720081 24220 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:53.720624 24220 heartbeater.cc:511] Master 127.13.162.62:46737 requested a full tablet report, sending...
I20260924 10:02:53.721911 23773 ts_manager.cc:194] Registered new tserver with Master: 839eb6300b2c465da3d1ee39a6acd2fe (127.13.162.3:39897)
I20260924 10:02:53.727322 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:02:53.733114 23961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:02:53.731549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.13.162.62:46737)
0924 10:02:53.731742 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:53.731749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:53.732256 (+ 507us) server_negotiation.cc:407] Connection header received
0924 10:02:53.732363 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:53.732367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:53.732440 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:53.732520 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:53.732848 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52874: BlockingRecv error: recv got EOF from 127.0.0.1:52874 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:02:53.733978 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23832
I20260924 10:02:53.737524 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.734946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.13.162.1:38179)
0924 10:02:53.735429 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:53.735436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:53.735461 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:02:53.735516 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:53.735522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:53.735736 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:53.735871 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:53.736843 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:53.736849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:53.736860 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:53.736877 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:53.736924 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:53.736938 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:53.737050 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:53.737188 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 10:02:53.737283 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260924 10:02:53.738817 23954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:53.816195 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23832
I20260924 10:02:53.829394 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23963
I20260924 10:02:53.836030 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.830261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.13.162.2:42787)
0924 10:02:53.830745 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:53.830752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:53.830777 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:02:53.830837 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:53.830843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:53.831096 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:53.831239 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:53.835328 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:53.835336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:53.835349 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:53.835366 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:53.835419 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:53.835439 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:53.835572 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:53.835671 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 10:02:53.835784 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
I20260924 10:02:53.837267 24085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:53.916178 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23963
I20260924 10:02:53.929268 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 24093
I20260924 10:02:53.933619 24233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:53.930244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.13.162.3:39897)
0924 10:02:53.931044 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:53.931052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:53.931088 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:02:53.931163 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:53.931171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:53.931463 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:53.931626 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:53.932775 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:53.932782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:53.932794 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:02:53.932810 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:53.932850 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:02:53.932863 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:53.933017 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:53.933199 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 10:02:53.933304 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":430,"threads_started":1}
I20260924 10:02:53.935294 24215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:54.023573 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 24093
I20260924 10:02:54.035774 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 23742
I20260924 10:02:54.036882 23803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:54.134997 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 23742
2026-09-24T10:02:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2215 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:54 dist-test-slave-jvsj krb5kdc[24238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:54 dist-test-slave-jvsj krb5kdc[24238](info): set up 2 sockets
Sep 24 10:02:54 dist-test-slave-jvsj krb5kdc[24238](info): commencing operation
krb5kdc: starting...
W20260924 10:02:55.135002 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:55 dist-test-slave-jvsj krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:02:55.347775 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35413
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:55.662390 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:55.662707 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:55.662833 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:55.670184 24251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:02:55.670305 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:55.670387 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:55.670463 24251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:02:55.670524 24251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:02:55.680550 24251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:55.681859 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:55.683182 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:55.694754 24257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:55.694833 24259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:55.694789 24256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:55.695474 24251 server_base.cc:1034] running on GCE node
I20260924 10:02:55.696256 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:55.697700 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:55.698895 24251 hybrid_clock.cc:648] HybridClock initialized: now 1790244175698844 us; error 40 us; skew 500 ppm
I20260924 10:02:55.699229 24251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:55 dist-test-slave-jvsj krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244175, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:55.702256 24251 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:02:55.703840 24251 webserver.cc:460] Webserver started at http://127.13.162.62:43373/ using document root <none> and password file <none>
I20260924 10:02:55.704516 24251 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:55.704629 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:55.704905 24251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:55.707894 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "4a12b251d3ae4203a233ea0ff39e3b81"
format_stamp: "Formatted at 2026-09-24 10:02:55 on dist-test-slave-jvsj"
server_key: "5635bb1b0de0827e798b8db8c99e2842"
server_key_iv: "65772a84ac260812ea0d7d1182444fda"
server_key_version: "encryptionkey@0"
I20260924 10:02:55.708663 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "4a12b251d3ae4203a233ea0ff39e3b81"
format_stamp: "Formatted at 2026-09-24 10:02:55 on dist-test-slave-jvsj"
server_key: "5635bb1b0de0827e798b8db8c99e2842"
server_key_iv: "65772a84ac260812ea0d7d1182444fda"
server_key_version: "encryptionkey@0"
I20260924 10:02:55.713994 24251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:55.717516 24266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:55.719000 24251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:55.719163 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "4a12b251d3ae4203a233ea0ff39e3b81"
format_stamp: "Formatted at 2026-09-24 10:02:55 on dist-test-slave-jvsj"
server_key: "5635bb1b0de0827e798b8db8c99e2842"
server_key_iv: "65772a84ac260812ea0d7d1182444fda"
server_key_version: "encryptionkey@0"
I20260924 10:02:55.719369 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:55.733798 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:55.737931 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:55.738206 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:55.755070 24251 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:35413
I20260924 10:02:55.755088 24317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:35413 every 8 connection(s)
I20260924 10:02:55.756417 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:02:55.757189 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 24251
I20260924 10:02:55.757440 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:02:55.758147 13960 external_mini_cluster.cc:1428] Setting key 7c1f913127caa85453a1a792e3b40268
I20260924 10:02:55.760727 24318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:02:55 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:55.773357 24318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81: Bootstrap starting.
I20260924 10:02:55.776491 24318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:55.777720 24318 log.cc:824] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:55.780536 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:55.760127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.13.162.62:35413)
0924 10:02:55.760873 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:55.760885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:55.760925 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:02:55.762472 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:55.762496 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:55.762887 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:55.763214 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:55.764924 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:55.766293 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:55.768013 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:55.768654 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:55.768772 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:55.773408 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:55.773415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:55.773433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:55.773486 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:55.776339 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:55.777098 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:55.777104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:55.777111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:55.777261 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:55.777641 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:55.777644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:55.777646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:55.778159 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:55.778503 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:55.778900 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 10:02:55.779309 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":312,"threads_started":1}
I20260924 10:02:55.780956 24318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81: No bootstrap required, opened a new log
I20260924 10:02:55.788491 24318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35413 } }
I20260924 10:02:55.788808 24318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:55.788884 24318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a12b251d3ae4203a233ea0ff39e3b81, State: Initialized, Role: FOLLOWER
I20260924 10:02:55.789386 24318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35413 } }
I20260924 10:02:55.789577 24318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:02:55.789669 24318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:02:55.789829 24318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:55.791879 24318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35413 } }
I20260924 10:02:55.792419 24318 leader_election.cc:302] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a12b251d3ae4203a233ea0ff39e3b81; no voters:
I20260924 10:02:55.792933 24318 leader_election.cc:288] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:02:55.793048 24323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:55.793628 24323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [term 1 LEADER]: Becoming Leader. State: Replica: 4a12b251d3ae4203a233ea0ff39e3b81, State: Running, Role: LEADER
I20260924 10:02:55.794150 24323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35413 } }
I20260924 10:02:55.794808 24318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:02:55.796743 24324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35413 } } }
I20260924 10:02:55.797106 24324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:55.798802 24325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a12b251d3ae4203a233ea0ff39e3b81. Latest consensus state: current_term: 1 leader_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a12b251d3ae4203a233ea0ff39e3b81" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35413 } } }
I20260924 10:02:55.799101 24325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81 [sys.catalog]: This master's current role is: LEADER
I20260924 10:02:55.799643 24332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:02:55.803731 24332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:02:55.811692 24332 catalog_manager.cc:1347] Generated new cluster ID: f11f438d65db4b988ee3c112678f09b8
I20260924 10:02:55.811908 24332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:02:55.827385 24332 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:02:55.828676 24332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:02:55.840255 24332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81: Generated new TSK 0
I20260924 10:02:55.841135 24332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:02:55.897488 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35413
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:56.214946 24344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:56.215222 24344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:56.215323 24344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:56.222450 24344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:56.222558 24344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:56.222704 24344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:02:56.232414 24344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:56.233706 24344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:56.235013 24344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:56.246654 24349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:56.247639 24350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:56.248255 24352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:56.248772 24344 server_base.cc:1034] running on GCE node
I20260924 10:02:56.249372 24344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:56.250308 24344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:56.251555 24344 hybrid_clock.cc:648] HybridClock initialized: now 1790244176251489 us; error 59 us; skew 500 ppm
I20260924 10:02:56.251856 24344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:56 dist-test-slave-jvsj krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244176, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:56.254881 24344 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:02:56.256227 24344 webserver.cc:460] Webserver started at http://127.13.162.1:36617/ using document root <none> and password file <none>
I20260924 10:02:56.256923 24344 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:56.257033 24344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:56.257296 24344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:56.260115 24344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "9cd10033290f494dab93f5c83b3a3d12"
format_stamp: "Formatted at 2026-09-24 10:02:56 on dist-test-slave-jvsj"
server_key: "07e3be9b204b9baa40f7e763f6902bba"
server_key_iv: "7d452c20b25d863dd7937e008c3f9f95"
server_key_version: "encryptionkey@0"
I20260924 10:02:56.260932 24344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd10033290f494dab93f5c83b3a3d12"
format_stamp: "Formatted at 2026-09-24 10:02:56 on dist-test-slave-jvsj"
server_key: "07e3be9b204b9baa40f7e763f6902bba"
server_key_iv: "7d452c20b25d863dd7937e008c3f9f95"
server_key_version: "encryptionkey@0"
I20260924 10:02:56.266042 24344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:56.269518 24359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:56.270908 24344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:02:56.271057 24344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "9cd10033290f494dab93f5c83b3a3d12"
format_stamp: "Formatted at 2026-09-24 10:02:56 on dist-test-slave-jvsj"
server_key: "07e3be9b204b9baa40f7e763f6902bba"
server_key_iv: "7d452c20b25d863dd7937e008c3f9f95"
server_key_version: "encryptionkey@0"
I20260924 10:02:56.271235 24344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:56.286217 24344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:56.290302 24344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:56.290607 24344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:56.291575 24344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:56.292878 24344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:56.292953 24344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:56.293069 24344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:56.293123 24344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:56.322372 24344 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:36131
I20260924 10:02:56.322422 24471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:36131 every 8 connection(s)
I20260924 10:02:56.323691 24344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:02:56.327543 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 24344
I20260924 10:02:56.327759 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:02:56.328509 13960 external_mini_cluster.cc:1428] Setting key 2dc994b10a61b1806addcd49dcba0190
Sep 24 10:02:56 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244176, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:56.344417 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:56.326116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:49669 (local address 127.13.162.62:35413)
0924 10:02:56.326598 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:56.326603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:56.329961 (+ 3358us) server_negotiation.cc:407] Connection header received
0924 10:02:56.330011 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:56.330014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:56.330074 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:56.330172 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:56.331824 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.332843 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.334466 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.334994 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.335111 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:56.338847 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:56.338851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:56.338855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:56.338887 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:56.341377 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:56.342179 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:56.342183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:56.342185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:56.342338 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:56.342767 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:56.342771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:56.342773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:56.343050 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:56.343176 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:56.343867 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 10:02:56.344081 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":294,"threads_started":1}
I20260924 10:02:56.345373 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:56.326654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35413 (local address 127.13.162.1:49669)
0924 10:02:56.327499 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:56.327554 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:56.329682 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:56.330550 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:56.330558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:56.331055 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:56.331633 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:56.331652 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.332996 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.333000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:56.334311 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:56.334352 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.335218 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.335223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:56.335475 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:56.336109 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:56.336132 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:56.338584 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:56.341514 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:56.341523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:56.341542 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:56.342076 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:56.342460 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:56.342465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:56.342469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:56.342655 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:56.343167 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:56.343175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:56.343505 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 10:02:56.343729 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 10:02:56.344244 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":321,"threads_started":1}
I20260924 10:02:56.347083 24472 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35413
I20260924 10:02:56.347368 24472 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:56.348060 24472 heartbeater.cc:511] Master 127.13.162.62:35413 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:02:56.350198 24283 ts_manager.cc:194] Registered new tserver with Master: 9cd10033290f494dab93f5c83b3a3d12 (127.13.162.1:36131)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:02:56.351826 24283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:49669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:02:56.375543 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35413
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:56.685779 24478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:56.686054 24478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:56.686168 24478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:56.693327 24478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:56.693437 24478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:56.693588 24478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:02:56.703554 24478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:56.704861 24478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:56.706192 24478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:56.718626 24483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:56.718797 24484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:56.718797 24486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:56.719580 24478 server_base.cc:1034] running on GCE node
I20260924 10:02:56.720173 24478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:56.721074 24478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:56.722280 24478 hybrid_clock.cc:648] HybridClock initialized: now 1790244176722224 us; error 38 us; skew 500 ppm
I20260924 10:02:56.722656 24478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:56 dist-test-slave-jvsj krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244176, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:56.725677 24478 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:02:56.727097 24478 webserver.cc:460] Webserver started at http://127.13.162.2:33971/ using document root <none> and password file <none>
I20260924 10:02:56.727886 24478 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:56.728000 24478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:56.728271 24478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:56.731289 24478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "dabbceca60c04517aa4c76e8b5cb04cf"
format_stamp: "Formatted at 2026-09-24 10:02:56 on dist-test-slave-jvsj"
server_key: "67133d81d39f931c31637cafa813d7a3"
server_key_iv: "6e3012c5f76be2137bd12dbc86135e75"
server_key_version: "encryptionkey@0"
I20260924 10:02:56.732012 24478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "dabbceca60c04517aa4c76e8b5cb04cf"
format_stamp: "Formatted at 2026-09-24 10:02:56 on dist-test-slave-jvsj"
server_key: "67133d81d39f931c31637cafa813d7a3"
server_key_iv: "6e3012c5f76be2137bd12dbc86135e75"
server_key_version: "encryptionkey@0"
I20260924 10:02:56.737051 24478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:02:56.740427 24493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:56.741779 24478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:02:56.741964 24478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "dabbceca60c04517aa4c76e8b5cb04cf"
format_stamp: "Formatted at 2026-09-24 10:02:56 on dist-test-slave-jvsj"
server_key: "67133d81d39f931c31637cafa813d7a3"
server_key_iv: "6e3012c5f76be2137bd12dbc86135e75"
server_key_version: "encryptionkey@0"
I20260924 10:02:56.742184 24478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:56.758872 24478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:56.762853 24478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:56.763129 24478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:56.764014 24478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:56.765277 24478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:56.765350 24478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:56.765460 24478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:56.765507 24478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:56.799189 24478 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:35511
I20260924 10:02:56.799225 24605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:35511 every 8 connection(s)
I20260924 10:02:56.800411 24478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:02:56.805204 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 24478
I20260924 10:02:56.805384 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:02:56.805773 13960 external_mini_cluster.cc:1428] Setting key 4d3917abf9b5b9361b4956858239fd89
Sep 24 10:02:56 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244176, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:56.820863 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:56.802766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:49709 (local address 127.13.162.62:35413)
0924 10:02:56.802925 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:56.802929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:56.804031 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 10:02:56.806003 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:56.806009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:56.806096 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:56.806207 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:56.807881 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.808892 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.810681 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.811210 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.811326 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:56.815006 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:56.815011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:56.815015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:56.815047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:56.817964 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:56.818716 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:56.818721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:56.818724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:56.818867 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:56.819256 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:56.819260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:56.819263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:56.819573 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:56.819711 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:56.820339 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 10:02:56.820558 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:02:56.821719 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:56.803192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35413 (local address 127.13.162.2:49709)
0924 10:02:56.803869 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:56.803921 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:56.805668 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:56.806521 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:56.806533 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:56.807048 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:56.807725 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:56.807744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.809073 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.809077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:56.810556 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:56.810568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:56.811467 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:56.811473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:56.811682 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:56.812323 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:56.812347 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:56.814764 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:56.818102 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:56.818110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:56.818125 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:56.818600 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:56.818980 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:56.818983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:56.818985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:56.819164 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:56.819674 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:56.819680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:56.820008 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 10:02:56.820225 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 10:02:56.820608 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":260,"threads_started":1}
I20260924 10:02:56.823182 24606 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35413
I20260924 10:02:56.823475 24606 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:56.824306 24606 heartbeater.cc:511] Master 127.13.162.62:35413 requested a full tablet report, sending...
I20260924 10:02:56.825753 24283 ts_manager.cc:194] Registered new tserver with Master: dabbceca60c04517aa4c76e8b5cb04cf (127.13.162.2:35511)
I20260924 10:02:56.826581 24283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:49709
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:02:56.853839 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35413
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:02:57.162703 24611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:02:57.162984 24611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:02:57.163110 24611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:02:57.170133 24611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:02:57.170239 24611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:02:57.170406 24611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:02:57.180543 24611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:02:57.181836 24611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:02:57.183117 24611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:02:57.195269 24616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:57.195230 24619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:02:57.195679 24617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:02:57.196077 24611 server_base.cc:1034] running on GCE node
I20260924 10:02:57.196753 24611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:02:57.197719 24611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:02:57.198969 24611 hybrid_clock.cc:648] HybridClock initialized: now 1790244177198905 us; error 41 us; skew 500 ppm
I20260924 10:02:57.199275 24611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:02:57 dist-test-slave-jvsj krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244177, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:02:57.202250 24611 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:02:57.203608 24611 webserver.cc:460] Webserver started at http://127.13.162.3:38349/ using document root <none> and password file <none>
I20260924 10:02:57.204216 24611 fs_manager.cc:359] Metadata directory not provided
I20260924 10:02:57.204305 24611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:02:57.204545 24611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:02:57.207322 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "f616b1ccc5084f09a613183b63d8c879"
format_stamp: "Formatted at 2026-09-24 10:02:57 on dist-test-slave-jvsj"
server_key: "c5e3309d0a2b1f98ef29f92455d07187"
server_key_iv: "60b0d80f589974429d64922fad8bbe43"
server_key_version: "encryptionkey@0"
I20260924 10:02:57.208014 24611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "f616b1ccc5084f09a613183b63d8c879"
format_stamp: "Formatted at 2026-09-24 10:02:57 on dist-test-slave-jvsj"
server_key: "c5e3309d0a2b1f98ef29f92455d07187"
server_key_iv: "60b0d80f589974429d64922fad8bbe43"
server_key_version: "encryptionkey@0"
I20260924 10:02:57.212831 24611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:02:57.216078 24626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:57.217365 24611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:02:57.217532 24611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "f616b1ccc5084f09a613183b63d8c879"
format_stamp: "Formatted at 2026-09-24 10:02:57 on dist-test-slave-jvsj"
server_key: "c5e3309d0a2b1f98ef29f92455d07187"
server_key_iv: "60b0d80f589974429d64922fad8bbe43"
server_key_version: "encryptionkey@0"
I20260924 10:02:57.217697 24611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:02:57.237674 24611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:02:57.241426 24611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:02:57.241704 24611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:02:57.242663 24611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:02:57.243942 24611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:02:57.244015 24611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:57.244139 24611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:02:57.244191 24611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:02:57.273041 24611 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:39031
I20260924 10:02:57.273082 24738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:39031 every 8 connection(s)
I20260924 10:02:57.274255 24611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:02:57.283308 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 24611
I20260924 10:02:57.283468 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:02:57.283859 13960 external_mini_cluster.cc:1428] Setting key efc91ab7200135b2c503d30e7ffa5bad
Sep 24 10:02:57 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244177, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:02:57.293282 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.276863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:40561 (local address 127.13.162.62:35413)
0924 10:02:57.277045 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.277050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.278159 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 10:02:57.279923 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.279927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.280008 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.280092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:57.281523 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.282234 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.283848 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.284391 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.284514 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.288029 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:57.288033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:57.288036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:57.288066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:57.290481 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.291131 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.291136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.291138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.291245 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.291635 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.291640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.291643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.291917 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:57.292045 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.292717 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.292948 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:02:57.294222 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.277295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35413 (local address 127.13.162.3:40561)
0924 10:02:57.277996 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:57.278036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:57.279679 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:57.280287 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:57.280295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:57.280823 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:02:57.281364 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.281381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.282393 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.282397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.283708 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.283720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.284622 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.284630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.284906 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.285525 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:02:57.285553 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:02:57.287840 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:02:57.290602 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:57.290611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:57.290628 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:57.291018 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:57.291339 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:02:57.291343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:02:57.291347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:02:57.291529 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:02:57.292030 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:02:57.292037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:02:57.292352 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 10:02:57.292592 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 10:02:57.293124 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":280,"threads_started":1}
I20260924 10:02:57.295617 24739 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35413
I20260924 10:02:57.295878 24739 heartbeater.cc:464] Registering TS with master...
I20260924 10:02:57.296557 24739 heartbeater.cc:511] Master 127.13.162.62:35413 requested a full tablet report, sending...
I20260924 10:02:57.298026 24282 ts_manager.cc:194] Registered new tserver with Master: f616b1ccc5084f09a613183b63d8c879 (127.13.162.3:39031)
I20260924 10:02:57.299075 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:02:57.299129 24282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:40561
I20260924 10:02:57.314772 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.303424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.13.162.62:35413)
0924 10:02:57.303603 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.303607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.304063 (+ 456us) server_negotiation.cc:407] Connection header received
0924 10:02:57.304192 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.304196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.304259 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.304330 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:57.305675 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.306436 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.307589 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.307976 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.308057 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.310083 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:57.310088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:57.310091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:57.310120 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:57.312314 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.312902 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.312906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.312908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.313018 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.313515 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.313518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.313520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.313804 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:57.313988 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.314397 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.314556 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:02:57.318495 24282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:02:57.325294 24282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:02:57.355247 24472 heartbeater.cc:503] Master 127.13.162.62:35413 was elected leader, sending a full tablet report...
I20260924 10:02:57.372846 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.336632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:35511 (local address 127.0.0.1:40794)
0924 10:02:57.364044 (+ 27412us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:57.364068 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:57.364212 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:57.365711 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:57.365722 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:57.365751 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:57.366160 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.366174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.368028 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.368034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.370357 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.370371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.372103 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.372109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.372292 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.372329 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:02:57.372450 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 10:02:57.372595 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27259,"thread_start_us":28998,"threads_started":1}
I20260924 10:02:57.373639 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.337155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.13.162.2:35511)
0924 10:02:57.363388 (+ 26233us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.363400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.364354 (+ 954us) server_negotiation.cc:407] Connection header received
0924 10:02:57.364475 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.364484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.364821 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.365029 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:57.366375 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.367884 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.370527 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.371905 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.372032 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.372232 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.373245 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.373376 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26049,"thread_start_us":26261,"threads_started":1}
I20260924 10:02:57.374229 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.337080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:36131 (local address 127.0.0.1:44352)
0924 10:02:57.365720 (+ 28640us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:57.365738 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:57.365983 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:57.366935 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:57.366940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:57.366968 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:57.367397 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.367415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.369219 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.369227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.371204 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.371218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.373446 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.373452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.373620 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.373651 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:02:57.373734 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:02:57.373856 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28494,"mutex_wait_us":102,"thread_start_us":25434,"threads_started":1}
I20260924 10:02:57.374711 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.336737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.13.162.3:39031)
0924 10:02:57.366382 (+ 29645us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.366390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.366417 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:02:57.366487 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.366492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.366657 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.366815 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:57.367447 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.369034 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.371405 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.373351 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.373477 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.373689 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.374000 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.374202 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29434,"thread_start_us":29269,"threads_started":1}
I20260924 10:02:57.375015 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.337671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.13.162.1:36131)
0924 10:02:57.361169 (+ 23498us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.361180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.365859 (+ 4679us) server_negotiation.cc:407] Connection header received
0924 10:02:57.366162 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.366170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.366473 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.366639 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:57.367619 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.369092 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.371544 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.372966 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.373095 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.373308 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.374517 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.374679 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23266,"thread_start_us":23522,"threads_started":1}
I20260924 10:02:57.375285 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.336447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:39031 (local address 127.0.0.1:34490)
0924 10:02:57.362001 (+ 25554us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:57.362053 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:57.362278 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:57.366825 (+ 4547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:57.366830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:57.366852 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:57.367263 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.367272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.369174 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.369179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.371214 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.371226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.373562 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.373569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.373811 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.373843 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:02:57.373933 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:02:57.374038 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25241,"mutex_wait_us":107,"thread_start_us":28993,"threads_started":1}
I20260924 10:02:57.378350 24674 tablet_service.cc:1467] Processing CreateTablet for tablet 3addd514fac343ffa9ee13652b6746b3 (DEFAULT_TABLE table=test-table [id=0ba108cc44e64baa88b3b75d2b27dfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:57.378261 24541 tablet_service.cc:1467] Processing CreateTablet for tablet 3addd514fac343ffa9ee13652b6746b3 (DEFAULT_TABLE table=test-table [id=0ba108cc44e64baa88b3b75d2b27dfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:57.378367 24407 tablet_service.cc:1467] Processing CreateTablet for tablet 3addd514fac343ffa9ee13652b6746b3 (DEFAULT_TABLE table=test-table [id=0ba108cc44e64baa88b3b75d2b27dfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:02:57.379532 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3addd514fac343ffa9ee13652b6746b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:57.379599 24541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3addd514fac343ffa9ee13652b6746b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:57.379599 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3addd514fac343ffa9ee13652b6746b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:02:57.391065 24757 tablet_bootstrap.cc:492] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Bootstrap starting.
I20260924 10:02:57.395040 24757 tablet_bootstrap.cc:654] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:57.396487 24757 log.cc:824] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:57.401217 24757 tablet_bootstrap.cc:492] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: No bootstrap required, opened a new log
I20260924 10:02:57.401482 24757 ts_tablet_manager.cc:1397] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 10:02:57.402490 24758 tablet_bootstrap.cc:492] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Bootstrap starting.
I20260924 10:02:57.406088 24758 tablet_bootstrap.cc:654] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:57.406947 24759 tablet_bootstrap.cc:492] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Bootstrap starting.
I20260924 10:02:57.407373 24758 log.cc:824] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:57.409726 24759 tablet_bootstrap.cc:654] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Neither blocks nor log segments found. Creating new log.
I20260924 10:02:57.410912 24759 log.cc:824] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Log is configured to *not* fsync() on all Append() calls
I20260924 10:02:57.411470 24757 raft_consensus.cc:348] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.411911 24757 raft_consensus.cc:374] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:57.412009 24757 raft_consensus.cc:729] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f616b1ccc5084f09a613183b63d8c879, State: Initialized, Role: FOLLOWER
I20260924 10:02:57.412627 24758 tablet_bootstrap.cc:492] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: No bootstrap required, opened a new log
I20260924 10:02:57.412730 24757 consensus_queue.cc:260] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.412915 24758 ts_tablet_manager.cc:1397] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260924 10:02:57.413806 24739 heartbeater.cc:503] Master 127.13.162.62:35413 was elected leader, sending a full tablet report...
I20260924 10:02:57.414250 24759 tablet_bootstrap.cc:492] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: No bootstrap required, opened a new log
I20260924 10:02:57.414474 24757 ts_tablet_manager.cc:1428] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 10:02:57.414506 24759 ts_tablet_manager.cc:1397] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:02:57.424009 24758 raft_consensus.cc:348] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.424492 24758 raft_consensus.cc:374] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:57.424312 24759 raft_consensus.cc:348] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.424602 24758 raft_consensus.cc:729] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd10033290f494dab93f5c83b3a3d12, State: Initialized, Role: FOLLOWER
I20260924 10:02:57.424728 24759 raft_consensus.cc:374] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:02:57.424813 24759 raft_consensus.cc:729] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabbceca60c04517aa4c76e8b5cb04cf, State: Initialized, Role: FOLLOWER
I20260924 10:02:57.425240 24758 consensus_queue.cc:260] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.425318 24759 consensus_queue.cc:260] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.426339 24606 heartbeater.cc:503] Master 127.13.162.62:35413 was elected leader, sending a full tablet report...
I20260924 10:02:57.427117 24759 ts_tablet_manager.cc:1428] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 10:02:57.427317 24758 ts_tablet_manager.cc:1428] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 10:02:57.457542 24764 raft_consensus.cc:482] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:02:57.457825 24764 raft_consensus.cc:504] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.459481 24764 leader_election.cc:288] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f616b1ccc5084f09a613183b63d8c879 (127.13.162.3:39031), dabbceca60c04517aa4c76e8b5cb04cf (127.13.162.2:35511)
I20260924 10:02:57.468186 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.459971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:53989 (local address 127.13.162.2:35511)
0924 10:02:57.460156 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.460162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.461082 (+ 920us) server_negotiation.cc:407] Connection header received
0924 10:02:57.461202 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.461207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.461271 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.461344 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:57.462070 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.463156 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.465895 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.467130 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.467279 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.467350 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.467798 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.467929 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:02:57.468186 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.459923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:40621 (local address 127.13.162.3:39031)
0924 10:02:57.460123 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.460129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.460822 (+ 693us) server_negotiation.cc:407] Connection header received
0924 10:02:57.461135 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.461141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.461216 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.461308 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:02:57.462066 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.463162 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.465900 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.467129 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.467286 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.467350 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.467797 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.467928 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:02:57.468331 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.459936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:35511 (local address 127.13.162.1:53989)
0924 10:02:57.460992 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:57.461006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:57.461105 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:57.461366 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:57.461370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:57.461386 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:57.461909 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.461923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.463288 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.463294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.465746 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.465762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.467392 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.467398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.467610 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.467668 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 10:02:57.467756 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:02:57.467937 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":159,"thread_start_us":746,"threads_started":1}
I20260924 10:02:57.468533 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.459923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:39031 (local address 127.13.162.1:40621)
0924 10:02:57.460697 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 10:02:57.460716 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:02:57.460963 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:02:57.461343 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:02:57.461348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:02:57.461376 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:02:57.461916 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.461924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.463288 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.463294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.465746 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:02:57.465762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.467358 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.467368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:02:57.467628 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.467671 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 10:02:57.467759 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:02:57.467947 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":228,"thread_start_us":420,"threads_started":1}
I20260924 10:02:57.469424 24561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3addd514fac343ffa9ee13652b6746b3" candidate_uuid: "9cd10033290f494dab93f5c83b3a3d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" is_pre_election: true
I20260924 10:02:57.469429 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3addd514fac343ffa9ee13652b6746b3" candidate_uuid: "9cd10033290f494dab93f5c83b3a3d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f616b1ccc5084f09a613183b63d8c879" is_pre_election: true
I20260924 10:02:57.469983 24561 raft_consensus.cc:2454] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd10033290f494dab93f5c83b3a3d12 in term 0.
I20260924 10:02:57.469983 24694 raft_consensus.cc:2454] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd10033290f494dab93f5c83b3a3d12 in term 0.
I20260924 10:02:57.470651 24362 leader_election.cc:302] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cd10033290f494dab93f5c83b3a3d12, f616b1ccc5084f09a613183b63d8c879; no voters:
I20260924 10:02:57.471028 24764 raft_consensus.cc:2781] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:02:57.471158 24764 raft_consensus.cc:482] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:02:57.471249 24764 raft_consensus.cc:3037] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:57.473357 24764 raft_consensus.cc:504] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.473945 24764 leader_election.cc:288] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [CANDIDATE]: Term 1 election: Requested vote from peers f616b1ccc5084f09a613183b63d8c879 (127.13.162.3:39031), dabbceca60c04517aa4c76e8b5cb04cf (127.13.162.2:35511)
I20260924 10:02:57.474406 24561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3addd514fac343ffa9ee13652b6746b3" candidate_uuid: "9cd10033290f494dab93f5c83b3a3d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabbceca60c04517aa4c76e8b5cb04cf"
I20260924 10:02:57.474393 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3addd514fac343ffa9ee13652b6746b3" candidate_uuid: "9cd10033290f494dab93f5c83b3a3d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f616b1ccc5084f09a613183b63d8c879"
I20260924 10:02:57.474612 24694 raft_consensus.cc:3037] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:57.474613 24561 raft_consensus.cc:3037] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:02:57.477895 24694 raft_consensus.cc:2454] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd10033290f494dab93f5c83b3a3d12 in term 1.
I20260924 10:02:57.477895 24561 raft_consensus.cc:2454] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd10033290f494dab93f5c83b3a3d12 in term 1.
I20260924 10:02:57.478432 24362 leader_election.cc:302] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9cd10033290f494dab93f5c83b3a3d12, dabbceca60c04517aa4c76e8b5cb04cf, f616b1ccc5084f09a613183b63d8c879; no voters:
I20260924 10:02:57.478710 24764 raft_consensus.cc:2781] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:02:57.479131 24764 raft_consensus.cc:686] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd10033290f494dab93f5c83b3a3d12, State: Running, Role: LEADER
I20260924 10:02:57.479611 24764 consensus_queue.cc:237] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } }
I20260924 10:02:57.484241 24282 catalog_manager.cc:5495] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd10033290f494dab93f5c83b3a3d12 (127.13.162.1). New cstate: current_term: 1 leader_uuid: "9cd10033290f494dab93f5c83b3a3d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cd10033290f494dab93f5c83b3a3d12" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 36131 } health_report { overall_health: HEALTHY } } }
I20260924 10:02:57.500483 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.495346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.13.162.1:36131)
0924 10:02:57.495578 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.495584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.495606 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:57.495843 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.495847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.495958 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.496097 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:02:57.496441 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.497161 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.498462 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.498856 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.498962 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.499423 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:02:57.499932 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:02:57.500061 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.500165 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.500244 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:02:57.508829 24694 raft_consensus.cc:1261] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 1 FOLLOWER]: Refusing update from remote peer 9cd10033290f494dab93f5c83b3a3d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:57.508829 24561 raft_consensus.cc:1261] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 1 FOLLOWER]: Refusing update from remote peer 9cd10033290f494dab93f5c83b3a3d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:02:57.509931 24764 consensus_queue.cc:1035] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f616b1ccc5084f09a613183b63d8c879" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:57.510229 24768 consensus_queue.cc:1035] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabbceca60c04517aa4c76e8b5cb04cf" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:02:57.521339 24771 mvcc.cc:205] Tried to move back new op lower bound from 7332840151064895488 to 7332840150959865856. Current Snapshot: MvccSnapshot[applied={T|T < 7332840151064895488}]
I20260924 10:02:57.540079 24282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:57.540374 24282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:02:57.543581 24282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a12b251d3ae4203a233ea0ff39e3b81: Sending DeleteTablet for 3 replicas of tablet 3addd514fac343ffa9ee13652b6746b3
I20260924 10:02:57.544601 24674 tablet_service.cc:1514] Processing DeleteTablet for tablet 3addd514fac343ffa9ee13652b6746b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:57 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:34490
I20260924 10:02:57.544700 24541 tablet_service.cc:1514] Processing DeleteTablet for tablet 3addd514fac343ffa9ee13652b6746b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:57 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:40794
I20260924 10:02:57.545362 24407 tablet_service.cc:1514] Processing DeleteTablet for tablet 3addd514fac343ffa9ee13652b6746b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:02:57 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:44352
I20260924 10:02:57.545722 24781 tablet_replica.cc:330] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: stopping tablet replica
I20260924 10:02:57.546084 24781 raft_consensus.cc:2229] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:57.546669 24781 raft_consensus.cc:2258] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:57.547271 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 24344
I20260924 10:02:57.549481 24781 ts_tablet_manager.cc:1905] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:57.554255 24781 ts_tablet_manager.cc:1918] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:57.554404 24781 log.cc:1196] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/3addd514fac343ffa9ee13652b6746b3
I20260924 10:02:57.554790 24781 ts_tablet_manager.cc:1939] T 3addd514fac343ffa9ee13652b6746b3 P f616b1ccc5084f09a613183b63d8c879: Deleting consensus metadata
I20260924 10:02:57.556174 24269 catalog_manager.cc:4841] TS f616b1ccc5084f09a613183b63d8c879 (127.13.162.3:39031): tablet 3addd514fac343ffa9ee13652b6746b3 (table test-table [id=0ba108cc44e64baa88b3b75d2b27dfef]) successfully deleted
I20260924 10:02:57.560031 24783 tablet_replica.cc:330] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: stopping tablet replica
I20260924 10:02:57.560223 24782 tablet_replica.cc:330] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: stopping tablet replica
I20260924 10:02:57.560461 24783 raft_consensus.cc:2229] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:02:57.560570 24782 raft_consensus.cc:2229] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:02:57.561036 24782 raft_consensus.cc:2258] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:57.561125 24783 raft_consensus.cc:2258] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:02:57.563131 24782 ts_tablet_manager.cc:1905] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:57.563145 24783 ts_tablet_manager.cc:1905] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:02:57.569774 24782 ts_tablet_manager.cc:1918] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:57.569936 24782 log.cc:1196] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/3addd514fac343ffa9ee13652b6746b3
I20260924 10:02:57.570487 24782 ts_tablet_manager.cc:1939] T 3addd514fac343ffa9ee13652b6746b3 P dabbceca60c04517aa4c76e8b5cb04cf: Deleting consensus metadata
I20260924 10:02:57.570561 24783 ts_tablet_manager.cc:1918] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:02:57.570732 24783 log.cc:1196] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/3addd514fac343ffa9ee13652b6746b3
Sep 24 10:02:57 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:02:57.571200 24783 ts_tablet_manager.cc:1939] T 3addd514fac343ffa9ee13652b6746b3 P 9cd10033290f494dab93f5c83b3a3d12: Deleting consensus metadata
I20260924 10:02:57.572465 24268 catalog_manager.cc:4841] TS dabbceca60c04517aa4c76e8b5cb04cf (127.13.162.2:35511): tablet 3addd514fac343ffa9ee13652b6746b3 (table test-table [id=0ba108cc44e64baa88b3b75d2b27dfef]) successfully deleted
I20260924 10:02:57.572824 24270 catalog_manager.cc:4841] TS 9cd10033290f494dab93f5c83b3a3d12 (127.13.162.1:36131): tablet 3addd514fac343ffa9ee13652b6746b3 (table test-table [id=0ba108cc44e64baa88b3b75d2b27dfef]) successfully deleted
I20260924 10:02:57.578812 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.548094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.13.162.1:36131)
0924 10:02:57.548249 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.548254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.560622 (+ 12368us) server_negotiation.cc:407] Connection header received
0924 10:02:57.560907 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.560912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.561007 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.561090 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:57.563041 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.564234 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.566057 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.566804 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.566900 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.572104 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:57.572118 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:57.572136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:57.572209 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:57.576063 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.576722 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.576732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.576735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.576901 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.577233 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.577238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.577241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.577687 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:57.577969 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.578189 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.578457 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:02:57.579465 24467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:57.675571 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 24344
I20260924 10:02:57.690137 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 24478
Sep 24 10:02:57 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:02:57.704614 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.690898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.13.162.2:35511)
0924 10:02:57.691078 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.691083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.691105 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:02:57.691292 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.691298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.691370 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.691451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:57.692637 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.693353 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.694605 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.695018 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.695126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.699070 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:57.699084 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:57.699100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:57.699133 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:57.701788 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.702693 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.702700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.702702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.702821 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.703163 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.703166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.703168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.703530 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:57.703794 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.704103 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.704303 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:02:57.705245 24601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:57.792369 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 24478
I20260924 10:02:57.806128 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 24611
Sep 24 10:02:57 dist-test-slave-jvsj krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:02:57.819949 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:02:57.806857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.13.162.3:39031)
0924 10:02:57.807034 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:02:57.807039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:02:57.807057 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:02:57.807198 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:02:57.807202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:02:57.807263 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:02:57.807342 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:02:57.808474 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.809207 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.810438 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:02:57.810846 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:02:57.810942 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:02:57.815012 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:02:57.815032 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:02:57.815054 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:02:57.815097 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:02:57.817556 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.818125 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.818132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.818135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.818269 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:02:57.818707 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:02:57.818711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:02:57.818713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:02:57.819086 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:02:57.819305 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 10:02:57.819541 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 10:02:57.819714 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:02:57.820597 24734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:57.910440 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 24611
I20260924 10:02:57.923218 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 24251
I20260924 10:02:57.924304 24313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:02:58.030143 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 24251
2026-09-24T10:02:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3896 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:02:58 dist-test-slave-jvsj krb5kdc[24791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:02:58 dist-test-slave-jvsj krb5kdc[24791](info): set up 2 sockets
Sep 24 10:02:58 dist-test-slave-jvsj krb5kdc[24791](info): commencing operation
krb5kdc: starting...
W20260924 10:02:59.034977 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:02:59 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:02:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/rangeradmin_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/rangeradmin_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/rangerlookup_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/rangerlookup_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
I20260924 10:02:59.329702 13960 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYd9YqT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/postgres -l logfile start
2026-09-24 10:03:01.679 UTC [24821] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:03:01.679 UTC [24821] LOG: listening on IPv4 address "127.13.162.20", port 60247
2026-09-24 10:03:01.682 UTC [24821] LOG: listening on Unix socket "/tmp/.s.PGSQL.60247"
2026-09-24 10:03:01.702 UTC [24824] LOG: database system was shut down at 2026-09-24 10:03:00 UTC
2026-09-24 10:03:01.706 UTC [24821] LOG: database system is ready to accept connections
I20260924 10:03:02.594702 13960 mini_postgres.cc:96] Postgres bound to 60247
2026-09-24 10:03:02.610 UTC [24832] FATAL: database "slave" does not exist
127.13.162.20:60247 - accepting connections
I20260924 10:03:02.611274 13960 mini_ranger.cc:161] Starting Ranger...
I20260924 10:03:02.635991 13960 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:03:03.101588 13960 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:03:03.101702 13960 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-admin
2026-09-24 10:03:03,372 [I] DB FLAVOR :POSTGRES
2026-09-24 10:03:03,372 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:03:03,372 [I] Checking connection..
2026-09-24 10:03:03,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:03:03,624 [I] Checking connection passed.
2026-09-24 10:03:03,625 [I] --------- Verifying version history table ---------
2026-09-24 10:03:03,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:03:03,866 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:03:03,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:03:04,119 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:03:04,119 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:03:04,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:03:04.368 UTC [24896] WARNING: there is no transaction in progress
2026-09-24 10:03:04,374 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:03:04,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:03:04,618 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:03:04,618 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:03:04,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:03:04,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:03:05,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jvsj.c.gcp-upstream.internal','N') ;"
2026-09-24 10:03:05,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:03:05,567 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:03:05,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:03:05,811 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:03:05,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:03:06,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:03:06,320 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:03:06,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:03:07.271 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.287 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.304 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.319 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.335 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.352 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.406 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.412 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.427 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.437 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.447 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.461 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.470 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.476 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.482 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:07.488 UTC [25004] WARNING: there is no transaction in progress
2026-09-24 10:03:08,083 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:03:08,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:03:08,322 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:03:08,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:03:08,566 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:03:08,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:03:08,800 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:03:08,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:03:09,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:03:09,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jvsj.c.gcp-upstream.internal';"
2026-09-24 10:03:09,556 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:03:09,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:03:09,798 [I] DB_PATCHES have already been applied
I20260924 10:03:09.803833 13960 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYd9YqT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:03:09.803926 13960 mini_ranger.cc:193] Using host: 127.13.162.20
I20260924 10:03:09.817526 13960 mini_ranger.cc:227] Ranger admin URL: http://127.13.162.20:60273
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:03:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:03:10.825302 13960 mini_ranger.cc:160] Time spent starting Ranger: real 8.214s user 0.004s sys 0.035s
Sep 24, 2026 10:03:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:03:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 10:03:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:03:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:03:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:03:31.012156 13960 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/rangerkms_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/rangerkms_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.162.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.162.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:03:31 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244211, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:03:31.174960 13960 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:03:31.193871 13960 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:03:32.023226 13960 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:03:32.023370 13960 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms
I20260924 10:03:32.023494 13960 mini_ranger_kms.cc:227] Using postgres at 127.13.162.20:60247
2026-09-24 10:03:32,412 [I] DB FLAVOR :POSTGRES
2026-09-24 10:03:32,412 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:03:32,412 [I] Checking connection
2026-09-24 10:03:32,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:03:32,667 [I] connection success
2026-09-24 10:03:32,667 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:03:32,667 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:03:32,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:03:32,906 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:03:32,906 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:03:32,906 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:03:32,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:60247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:03:33,160 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:03:33.595443 13960 mini_ranger_kms.cc:310] Created kms service
I20260924 10:03:33.770473 13960 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:03:33.812107 13960 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:03:33 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244211, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:03:34.211442 13960 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:03:34.211736 13960 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:03:34.211796 13960 mini_ranger_kms.cc:240] Using host: 127.13.162.20
I20260924 10:03:34.228721 13960 mini_ranger_kms.cc:276] Ranger KMS PID: 25184
I20260924 10:03:34.228988 13960 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.162.20:35775
Sep 24, 2026 10:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:03:36.233938 13960 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.059s user 0.117s sys 0.291s
I20260924 10:03:36.234200 13960 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:03:36.234453 13960 mini_ranger_kms.cc:401] 127.13.162.20:35775/kms/v1/keys
Sep 24 10:03:36 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.162.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:03:36 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.162.20@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
Sep 24 10:03:37 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.162.20@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
Sep 24 10:03:39 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:03:39.310217 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:41425
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:41425
--ranger_config_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:35775/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:03:39.637020 25235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:03:39.637320 25235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:03:39.637405 25235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:03:39.644626 25235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:03:39.644750 25235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:03:39.644794 25235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:03:39.644851 25235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:03:39.644896 25235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:03:39.655117 25235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.162.20:35775/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:41425
--ranger_config_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:41425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:03:39.656373 25235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:03:39.657629 25235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:03:39.668962 25243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:39.669042 25241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:39.669091 25240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:03:39.669725 25235 server_base.cc:1034] running on GCE node
I20260924 10:03:39.670573 25235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:03:39.671986 25235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:03:39.673207 25235 hybrid_clock.cc:648] HybridClock initialized: now 1790244219673164 us; error 33 us; skew 500 ppm
I20260924 10:03:39.673540 25235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:03:39 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:03:39.680794 25235 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:03:39.682410 25235 webserver.cc:460] Webserver started at http://127.13.162.62:46533/ using document root <none> and password file <none>
I20260924 10:03:39.683068 25235 fs_manager.cc:359] Metadata directory not provided
I20260924 10:03:39.683166 25235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:03:39.683430 25235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:03:39 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:03:39.747889 25235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "bb95129504b243a7ad0d7991ca175d7c"
format_stamp: "Formatted at 2026-09-24 10:03:39 on dist-test-slave-jvsj"
server_key: "eea97fb7a3671bd63639e0b42a68daec"
server_key_iv: "7c15b755c5092e6b5c49c1214df84c73"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:39.748961 25235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "bb95129504b243a7ad0d7991ca175d7c"
format_stamp: "Formatted at 2026-09-24 10:03:39 on dist-test-slave-jvsj"
server_key: "eea97fb7a3671bd63639e0b42a68daec"
server_key_iv: "7c15b755c5092e6b5c49c1214df84c73"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:39.755017 25235 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260924 10:03:39.775707 25250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:39.777645 25235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:03:39.777848 25235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "bb95129504b243a7ad0d7991ca175d7c"
format_stamp: "Formatted at 2026-09-24 10:03:39 on dist-test-slave-jvsj"
server_key: "eea97fb7a3671bd63639e0b42a68daec"
server_key_iv: "7c15b755c5092e6b5c49c1214df84c73"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:39.778107 25235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:03:39.802342 25235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:03:39.807638 25235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:03:39.807955 25235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:03:39.826578 25235 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:41425
I20260924 10:03:39.826570 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:41425 every 8 connection(s)
I20260924 10:03:39.828447 25235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:03:39.832530 25302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:03:39.835799 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 25235
I20260924 10:03:39.836010 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:03:39.836812 13960 external_mini_cluster.cc:1428] Setting key c483559d894d31fc1c13ca9e0042f0c6
I20260924 10:03:39.846359 25302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c: Bootstrap starting.
I20260924 10:03:39.851310 25302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c: Neither blocks nor log segments found. Creating new log.
I20260924 10:03:39.852993 25302 log.cc:824] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:03:39.856412 25302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c: No bootstrap required, opened a new log
I20260924 10:03:39.866005 25302 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb95129504b243a7ad0d7991ca175d7c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41425 } }
I20260924 10:03:39.866425 25302 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:03:39.866560 25302 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb95129504b243a7ad0d7991ca175d7c, State: Initialized, Role: FOLLOWER
I20260924 10:03:39.867208 25302 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb95129504b243a7ad0d7991ca175d7c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41425 } }
I20260924 10:03:39.867434 25302 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:03:39.867591 25302 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:03:39.867755 25302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:03:39 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:03:39.870280 25302 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb95129504b243a7ad0d7991ca175d7c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41425 } }
I20260924 10:03:39.870975 25302 leader_election.cc:302] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb95129504b243a7ad0d7991ca175d7c; no voters:
I20260924 10:03:39.871767 25302 leader_election.cc:288] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:03:39.872196 25307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:03:39.876253 25302 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:03:39.879348 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:39.839111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.13.162.62:41425)
0924 10:03:39.853772 (+ 14661us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:39.853791 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:39.853849 (+ 58us) server_negotiation.cc:407] Connection header received
0924 10:03:39.856526 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:39.856583 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:39.857171 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:39.857658 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:39.859309 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:39.861570 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:39.863527 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:39.864674 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:39.864865 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:39.869756 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:39.869767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:39.869791 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:39.869849 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:39.874129 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:39.875089 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:39.875098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:39.875107 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:39.875294 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:39.875762 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:39.875766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:39.875769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:39.876533 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:39.876842 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:39.877313 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 10:03:39.877805 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14372,"thread_start_us":14711,"threads_started":1}
I20260924 10:03:39.881474 25307 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [term 1 LEADER]: Becoming Leader. State: Replica: bb95129504b243a7ad0d7991ca175d7c, State: Running, Role: LEADER
I20260924 10:03:39.882139 25307 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb95129504b243a7ad0d7991ca175d7c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41425 } }
I20260924 10:03:39.892031 25309 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb95129504b243a7ad0d7991ca175d7c. Latest consensus state: current_term: 1 leader_uuid: "bb95129504b243a7ad0d7991ca175d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb95129504b243a7ad0d7991ca175d7c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41425 } } }
I20260924 10:03:39.892597 25309 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [sys.catalog]: This master's current role is: LEADER
I20260924 10:03:39.893180 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb95129504b243a7ad0d7991ca175d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb95129504b243a7ad0d7991ca175d7c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41425 } } }
I20260924 10:03:39.893527 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c [sys.catalog]: This master's current role is: LEADER
I20260924 10:03:39.896936 25314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:03:39.902838 25314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:03:39.920467 25314 catalog_manager.cc:1347] Generated new cluster ID: 4f2c6aed888a435494770523021920e8
I20260924 10:03:39.920683 25314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:03:39.931553 25302 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:03:39.932528 25314 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:03:39.934481 25314 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:03:39.947954 25314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c: Generated new TSK 0
I20260924 10:03:39.949270 25314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:03:40.775758 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:35775/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41425
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:03:41.145108 25347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:03:41.145433 25347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:03:41.145563 25347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:03:41.153172 25347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:03:41.153306 25347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:03:41.153486 25347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:03:41.164009 25347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.162.20:35775/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:03:41.165347 25347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:03:41.166702 25347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:03:41.185290 25357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:41.185863 25358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:41.190644 25360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:03:41.191218 25347 server_base.cc:1034] running on GCE node
I20260924 10:03:41.191859 25347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:03:41.192893 25347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:03:41.194145 25347 hybrid_clock.cc:648] HybridClock initialized: now 1790244221194075 us; error 56 us; skew 500 ppm
I20260924 10:03:41.194546 25347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:03:41.198196 25347 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:03:41.202236 25347 webserver.cc:460] Webserver started at http://127.13.162.1:41501/ using document root <none> and password file <none>
I20260924 10:03:41.202965 25347 fs_manager.cc:359] Metadata directory not provided
I20260924 10:03:41.203084 25347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:03:41.203397 25347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:03:41.225867 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "a621a2595e42484ab03383c6dfff62ab"
format_stamp: "Formatted at 2026-09-24 10:03:41 on dist-test-slave-jvsj"
server_key: "d04e2960140b62d1d1eb3074403a8aaf"
server_key_iv: "e1ce8dcdb1b584df6d0b1cefedc223bc"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:41.229476 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "a621a2595e42484ab03383c6dfff62ab"
format_stamp: "Formatted at 2026-09-24 10:03:41 on dist-test-slave-jvsj"
server_key: "d04e2960140b62d1d1eb3074403a8aaf"
server_key_iv: "e1ce8dcdb1b584df6d0b1cefedc223bc"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:41.236461 25347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 10:03:41.270308 25367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:41.271993 25347 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.000s
I20260924 10:03:41.272183 25347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "a621a2595e42484ab03383c6dfff62ab"
format_stamp: "Formatted at 2026-09-24 10:03:41 on dist-test-slave-jvsj"
server_key: "d04e2960140b62d1d1eb3074403a8aaf"
server_key_iv: "e1ce8dcdb1b584df6d0b1cefedc223bc"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:41.272413 25347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:03:41.310931 25347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:03:41.315776 25347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:03:41.316200 25347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:03:41.317515 25347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:03:41.321687 25347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:03:41.321798 25347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:41.321971 25347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:03:41.322043 25347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:41.370298 25347 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:42675
I20260924 10:03:41.370342 25479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:42675 every 8 connection(s)
I20260924 10:03:41.372304 25347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:03:41.381973 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 25347
I20260924 10:03:41.382205 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:03:41.383035 13960 external_mini_cluster.cc:1428] Setting key fa64034a3e2148fbfbc11a5e6a10a085
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:03:41.398181 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:41.375236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:42723 (local address 127.13.162.62:41425)
0924 10:03:41.375665 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:41.375669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:41.377191 (+ 1522us) server_negotiation.cc:407] Connection header received
0924 10:03:41.379125 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:41.379131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:41.379224 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:41.379325 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:41.380763 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:41.381534 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:41.384099 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:41.384758 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:41.384856 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:41.390563 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:41.390572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:41.390577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:41.390619 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:41.394275 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:41.395333 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:41.395338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:41.395341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:41.395492 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:41.396048 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:41.396053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:41.396056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:41.396422 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:41.396554 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:41.397561 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0924 10:03:41.397803 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260924 10:03:41.400777 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:41.376078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:41425 (local address 127.13.162.1:42723)
0924 10:03:41.377042 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:41.377081 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:41.378853 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:41.379582 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:41.379590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:41.380024 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:03:41.380599 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:41.380616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:41.381711 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:41.381716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:41.383923 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:41.383950 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:41.385182 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:41.385189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:41.385501 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:41.387395 (+ 1894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:03:41.387426 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:03:41.390238 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:03:41.394507 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:03:41.394516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:03:41.394539 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:03:41.395163 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:03:41.395640 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:03:41.395644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:03:41.395648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:03:41.395914 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:03:41.396645 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:03:41.396653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:03:41.397120 (+ 467us) client_negotiation.cc:769] Sending connection context
0924 10:03:41.398762 (+ 1642us) client_negotiation.cc:240] Negotiation successful
0924 10:03:41.399348 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":309,"threads_started":1}
I20260924 10:03:41.402647 25480 heartbeater.cc:347] Connected to a master server at 127.13.162.62:41425
I20260924 10:03:41.403038 25480 heartbeater.cc:464] Registering TS with master...
I20260924 10:03:41.404013 25480 heartbeater.cc:511] Master 127.13.162.62:41425 requested a full tablet report, sending...
I20260924 10:03:41.406672 25267 ts_manager.cc:194] Registered new tserver with Master: a621a2595e42484ab03383c6dfff62ab (127.13.162.1:42675)
I20260924 10:03:41.408697 25267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:42723
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:03:41.445598 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:35775/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41425
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
W20260924 10:03:41.877537 25486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:03:41.877987 25486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:03:41.878152 25486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
W20260924 10:03:41.887428 25486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:03:41.887624 25486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:03:41.887853 25486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:03:41.902077 25486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.162.20:35775/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:03:41.903908 25486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:03:41.905617 25486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:03:41.938627 25492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:41.940685 25493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:03:41.953591 25486 server_base.cc:1034] running on GCE node
W20260924 10:03:41.954080 25496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:03:41.955039 25486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:03:41.956377 25486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:03:41.957619 25486 hybrid_clock.cc:648] HybridClock initialized: now 1790244221957481 us; error 131 us; skew 500 ppm
I20260924 10:03:41.958029 25486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:03:41.962242 25486 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:03:41.990804 25486 webserver.cc:460] Webserver started at http://127.13.162.2:45887/ using document root <none> and password file <none>
I20260924 10:03:41.992023 25486 fs_manager.cc:359] Metadata directory not provided
I20260924 10:03:41.992290 25486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:03:41.992723 25486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:03:41 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:03:42.030551 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "3c7531814a33444b9db29f4a4239f27c"
format_stamp: "Formatted at 2026-09-24 10:03:42 on dist-test-slave-jvsj"
server_key: "23c152d6b6262d170d35d843b08d58a5"
server_key_iv: "67f58507e380190dac2563745d4cea23"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:42.031834 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "3c7531814a33444b9db29f4a4239f27c"
format_stamp: "Formatted at 2026-09-24 10:03:42 on dist-test-slave-jvsj"
server_key: "23c152d6b6262d170d35d843b08d58a5"
server_key_iv: "67f58507e380190dac2563745d4cea23"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:42.045910 25486 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260924 10:03:42.096649 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:42.100528 25486 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.006s
I20260924 10:03:42.100839 25486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "3c7531814a33444b9db29f4a4239f27c"
format_stamp: "Formatted at 2026-09-24 10:03:42 on dist-test-slave-jvsj"
server_key: "23c152d6b6262d170d35d843b08d58a5"
server_key_iv: "67f58507e380190dac2563745d4cea23"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:42.101231 25486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:03:42.133625 25486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:03:42.139101 25486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:03:42.139457 25486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:03:42.140702 25486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:03:42.142500 25486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:03:42.142593 25486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:42.142765 25486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:03:42.142838 25486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:42.196285 25486 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:44637
I20260924 10:03:42.196301 25622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:44637 every 8 connection(s)
I20260924 10:03:42.198112 25486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:03:42.199692 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 25486
I20260924 10:03:42.199848 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:03:42.200249 13960 external_mini_cluster.cc:1428] Setting key 09eb78fc9c0c073d271ff2699aa7728f
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Sep 24 10:03:42 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:03:42.239665 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.202076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:46987 (local address 127.13.162.62:41425)
0924 10:03:42.218773 (+ 16697us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.218781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.219607 (+ 826us) server_negotiation.cc:407] Connection header received
0924 10:03:42.222527 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.222534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.222640 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.222742 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:42.224957 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.225685 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.227345 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.227722 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.227839 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.232933 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:42.232940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:42.232945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:42.232987 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:42.236264 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:42.237230 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:42.237235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:42.237239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:42.237378 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:42.237879 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:42.237884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:42.237887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:42.238206 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:42.238367 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.239083 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.239307 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16547,"thread_start_us":16412,"threads_started":1}
I20260924 10:03:42.240702 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.202921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:41425 (local address 127.13.162.2:46987)
0924 10:03:42.219382 (+ 16461us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.219450 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.221993 (+ 2543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.223221 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.223234 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.223859 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:03:42.224735 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.224773 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.225850 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.225853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.227187 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.227210 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.227939 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.227945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.228173 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.228793 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:03:42.228816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:03:42.232645 (+ 3829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:03:42.236434 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:03:42.236444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:03:42.236464 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:03:42.237095 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:03:42.237488 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:03:42.237493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:03:42.237497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:03:42.237750 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:03:42.238368 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:03:42.238377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:03:42.238718 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.238960 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.239507 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16123,"thread_start_us":15882,"threads_started":1}
I20260924 10:03:42.242044 25623 heartbeater.cc:347] Connected to a master server at 127.13.162.62:41425
I20260924 10:03:42.242389 25623 heartbeater.cc:464] Registering TS with master...
I20260924 10:03:42.243088 25623 heartbeater.cc:511] Master 127.13.162.62:41425 requested a full tablet report, sending...
I20260924 10:03:42.244457 25267 ts_manager.cc:194] Registered new tserver with Master: 3c7531814a33444b9db29f4a4239f27c (127.13.162.2:44637)
I20260924 10:03:42.245229 25267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:46987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:03:42.262068 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:35775/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41425
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:03:42.412981 25480 heartbeater.cc:503] Master 127.13.162.62:41425 was elected leader, sending a full tablet report...
W20260924 10:03:42.589924 25629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:03:42.590224 25629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:03:42.590392 25629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:03:42.597656 25629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:03:42.597769 25629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:03:42.597913 25629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:03:42.607990 25629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.162.20:35775/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:03:42.609287 25629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:03:42.610625 25629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:03:42.623594 25634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:42.623564 25637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:03:42.623597 25635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:03:42.624300 25629 server_base.cc:1034] running on GCE node
I20260924 10:03:42.624863 25629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:03:42.625766 25629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:03:42.626955 25629 hybrid_clock.cc:648] HybridClock initialized: now 1790244222626898 us; error 48 us; skew 500 ppm
I20260924 10:03:42.627260 25629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:03:42 dist-test-slave-jvsj krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:03:42.630213 25629 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:03:42.631601 25629 webserver.cc:460] Webserver started at http://127.13.162.3:33701/ using document root <none> and password file <none>
I20260924 10:03:42.632220 25629 fs_manager.cc:359] Metadata directory not provided
I20260924 10:03:42.632311 25629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:03:42.632552 25629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:03:42 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:03:42.649145 25629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "a843f1d594a84c90be81d68deeeb52f9"
format_stamp: "Formatted at 2026-09-24 10:03:42 on dist-test-slave-jvsj"
server_key: "95f278d18ea4465e4086dac0b683c4f9"
server_key_iv: "1af084f446886f622b2f91fa90f501dc"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:42.650254 25629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "a843f1d594a84c90be81d68deeeb52f9"
format_stamp: "Formatted at 2026-09-24 10:03:42 on dist-test-slave-jvsj"
server_key: "95f278d18ea4465e4086dac0b683c4f9"
server_key_iv: "1af084f446886f622b2f91fa90f501dc"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:42.657262 25629 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 10:03:42.672801 25644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:42.674304 25629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:03:42.674520 25629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "a843f1d594a84c90be81d68deeeb52f9"
format_stamp: "Formatted at 2026-09-24 10:03:42 on dist-test-slave-jvsj"
server_key: "95f278d18ea4465e4086dac0b683c4f9"
server_key_iv: "1af084f446886f622b2f91fa90f501dc"
server_key_version: "kuduclusterkey@0"
I20260924 10:03:42.674726 25629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:03:42.697943 25629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:03:42.702003 25629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:03:42.702275 25629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:03:42.703207 25629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:03:42.704577 25629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:03:42.704649 25629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:42.704754 25629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:03:42.704799 25629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:03:42.734225 25629 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:41251
I20260924 10:03:42.734261 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:41251 every 8 connection(s)
I20260924 10:03:42.735514 25629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:03:42.736387 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 25629
I20260924 10:03:42.736552 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:03:42.737073 13960 external_mini_cluster.cc:1428] Setting key bfd852fba48e6c746aacf0ea9ca9eed3
Sep 24 10:03:42 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:03:42.754813 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.738133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:35007 (local address 127.13.162.62:41425)
0924 10:03:42.738295 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.738301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.739744 (+ 1443us) server_negotiation.cc:407] Connection header received
0924 10:03:42.741761 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.741765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.741843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.741922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:42.743357 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.744041 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.745548 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.745947 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.746053 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.749252 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:42.749257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:42.749260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:42.749290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:42.751951 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:42.752624 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:42.752628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:42.752630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:42.752745 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:42.753168 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:42.753173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:42.753175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:42.753443 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:42.753570 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.754239 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.754492 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:03:42.755756 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.738837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:41425 (local address 127.13.162.3:35007)
0924 10:03:42.739606 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.739649 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.741306 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.742169 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.742178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.742641 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:03:42.743207 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.743223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.744182 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.744186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.745411 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.745423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.746155 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.746162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.746419 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.747030 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:03:42.747055 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:03:42.749054 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:03:42.752078 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:03:42.752087 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:03:42.752106 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:03:42.752510 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:03:42.752843 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:03:42.752848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:03:42.752852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:03:42.753060 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:03:42.753556 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:03:42.753565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:03:42.753885 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.754126 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.754654 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":309,"threads_started":1}
I20260924 10:03:42.757145 25757 heartbeater.cc:347] Connected to a master server at 127.13.162.62:41425
I20260924 10:03:42.757436 25757 heartbeater.cc:464] Registering TS with master...
I20260924 10:03:42.758127 25757 heartbeater.cc:511] Master 127.13.162.62:41425 requested a full tablet report, sending...
I20260924 10:03:42.759544 25267 ts_manager.cc:194] Registered new tserver with Master: a843f1d594a84c90be81d68deeeb52f9 (127.13.162.3:41251)
I20260924 10:03:42.760304 25267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:35007
I20260924 10:03:42.764865 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:03:42.781210 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.769190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.13.162.62:41425)
0924 10:03:42.769336 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.769340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.769827 (+ 487us) server_negotiation.cc:407] Connection header received
0924 10:03:42.769966 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.769970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.770041 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.770125 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:42.771507 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.772201 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.773586 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.773985 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.774071 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.776330 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:42.776335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:42.776338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:42.776375 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:42.778867 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:42.779678 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:42.779683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:42.779686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:42.779832 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:42.780156 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:42.780159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:42.780161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:42.780388 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:42.780603 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.780819 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.780967 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:03:42.784971 25267 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:03:42.791887 25267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:03:42.815802 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.803923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41251 (local address 127.0.0.1:37420)
0924 10:03:42.805918 (+ 1995us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.805965 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.806148 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.807363 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.807371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.807404 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:03:42.807888 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.807903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.810147 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.810153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.812978 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.812999 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.814773 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.814779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.815020 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.815100 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.815237 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.815404 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":5,"thread_start_us":1480,"threads_started":1}
I20260924 10:03:42.816517 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.804208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.13.162.3:41251)
0924 10:03:42.805379 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.805389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.806372 (+ 983us) server_negotiation.cc:407] Connection header received
0924 10:03:42.806490 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.806498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.806729 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.806939 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:03:42.808079 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.809863 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.813221 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.814473 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.814594 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.814841 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.816060 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.816211 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":851,"threads_started":1}
I20260924 10:03:42.820996 25692 tablet_service.cc:1467] Processing CreateTablet for tablet e2d49859e2ef4a79b4e2f12fd7b9139b (DEFAULT_TABLE table=test-table [id=2bc7c14d84f44f45bca1c0526c34f9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:03:42.821779 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.804787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.13.162.2:44637)
0924 10:03:42.805770 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.805778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.813044 (+ 7266us) server_negotiation.cc:407] Connection header received
0924 10:03:42.813524 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.813534 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.813843 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.814031 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:03:42.815049 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.816534 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.818979 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.820325 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.821045 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.821224 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.821311 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.821477 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":717,"threads_started":1}
I20260924 10:03:42.822022 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.804170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44637 (local address 127.0.0.1:49406)
0924 10:03:42.812887 (+ 8717us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.812921 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.813303 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.814297 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.814302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.814353 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:03:42.814867 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.814880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.816692 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.816698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.818800 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.818825 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.820555 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.820561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.820722 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.820757 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.820819 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.820959 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8574,"mutex_wait_us":21,"spinlock_wait_cycles":6912,"thread_start_us":8406,"threads_started":1}
I20260924 10:03:42.822042 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.806931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.13.162.1:42675)
0924 10:03:42.813833 (+ 6902us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.813841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.813870 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:03:42.813985 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.813992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.814215 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.814449 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:03:42.815556 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.817570 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.819644 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.820623 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.820735 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.820968 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.821587 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.821750 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6737,"thread_start_us":6611,"threads_started":1}
I20260924 10:03:42.822504 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.807017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:42675 (local address 127.0.0.1:55214)
0924 10:03:42.813162 (+ 6145us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.813176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.813302 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.814449 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.814457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.814486 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:03:42.815133 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.815146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.817736 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.817741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.819499 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.819511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.821103 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.821109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.821259 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.821288 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.821367 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.821485 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6029,"mutex_wait_us":101,"spinlock_wait_cycles":2560,"thread_start_us":6051,"threads_started":1}
I20260924 10:03:42.823702 25692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d49859e2ef4a79b4e2f12fd7b9139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:03:42.825340 25415 tablet_service.cc:1467] Processing CreateTablet for tablet e2d49859e2ef4a79b4e2f12fd7b9139b (DEFAULT_TABLE table=test-table [id=2bc7c14d84f44f45bca1c0526c34f9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:03:42.825551 25558 tablet_service.cc:1467] Processing CreateTablet for tablet e2d49859e2ef4a79b4e2f12fd7b9139b (DEFAULT_TABLE table=test-table [id=2bc7c14d84f44f45bca1c0526c34f9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:03:42.826704 25415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d49859e2ef4a79b4e2f12fd7b9139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:03:42.826828 25558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d49859e2ef4a79b4e2f12fd7b9139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:03:42.838070 25775 tablet_bootstrap.cc:492] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Bootstrap starting.
I20260924 10:03:42.843500 25775 tablet_bootstrap.cc:654] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Neither blocks nor log segments found. Creating new log.
I20260924 10:03:42.845057 25775 log.cc:824] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:03:42.846410 25776 tablet_bootstrap.cc:492] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Bootstrap starting.
I20260924 10:03:42.848158 25777 tablet_bootstrap.cc:492] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Bootstrap starting.
I20260924 10:03:42.851223 25776 tablet_bootstrap.cc:654] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Neither blocks nor log segments found. Creating new log.
I20260924 10:03:42.852303 25777 tablet_bootstrap.cc:654] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Neither blocks nor log segments found. Creating new log.
I20260924 10:03:42.852517 25776 log.cc:824] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:03:42.854429 25777 log.cc:824] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Log is configured to *not* fsync() on all Append() calls
I20260924 10:03:42.855106 25776 tablet_bootstrap.cc:492] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: No bootstrap required, opened a new log
I20260924 10:03:42.855206 25775 tablet_bootstrap.cc:492] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: No bootstrap required, opened a new log
I20260924 10:03:42.855448 25776 ts_tablet_manager.cc:1397] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 10:03:42.855460 25775 ts_tablet_manager.cc:1397] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260924 10:03:42.857466 25777 tablet_bootstrap.cc:492] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: No bootstrap required, opened a new log
I20260924 10:03:42.857769 25777 ts_tablet_manager.cc:1397] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 10:03:42.862865 25776 raft_consensus.cc:348] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.863324 25776 raft_consensus.cc:374] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:03:42.863466 25776 raft_consensus.cc:729] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7531814a33444b9db29f4a4239f27c, State: Initialized, Role: FOLLOWER
I20260924 10:03:42.864080 25776 consensus_queue.cc:260] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.865427 25623 heartbeater.cc:503] Master 127.13.162.62:41425 was elected leader, sending a full tablet report...
I20260924 10:03:42.865736 25775 raft_consensus.cc:348] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.866263 25775 raft_consensus.cc:374] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:03:42.866425 25775 raft_consensus.cc:729] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a843f1d594a84c90be81d68deeeb52f9, State: Initialized, Role: FOLLOWER
I20260924 10:03:42.867218 25775 consensus_queue.cc:260] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.867512 25776 ts_tablet_manager.cc:1428] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260924 10:03:42.868151 25777 raft_consensus.cc:348] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.868620 25777 raft_consensus.cc:374] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:03:42.868713 25777 raft_consensus.cc:729] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a621a2595e42484ab03383c6dfff62ab, State: Initialized, Role: FOLLOWER
I20260924 10:03:42.869383 25777 consensus_queue.cc:260] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.871706 25777 ts_tablet_manager.cc:1428] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260924 10:03:42.874468 25757 heartbeater.cc:503] Master 127.13.162.62:41425 was elected leader, sending a full tablet report...
W20260924 10:03:42.874878 25481 tablet.cc:2369] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:03:42.875523 25775 ts_tablet_manager.cc:1428] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260924 10:03:42.945855 25783 raft_consensus.cc:482] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:03:42.946148 25783 raft_consensus.cc:504] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.947945 25783 leader_election.cc:288] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a843f1d594a84c90be81d68deeeb52f9 (127.13.162.3:41251), 3c7531814a33444b9db29f4a4239f27c (127.13.162.2:44637)
W20260924 10:03:42.950117 25624 tablet.cc:2369] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:03:42.955899 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.948514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52323 (local address 127.13.162.2:44637)
0924 10:03:42.948722 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.948728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.949349 (+ 621us) server_negotiation.cc:407] Connection header received
0924 10:03:42.949519 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.949523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.949585 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.949654 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:03:42.950273 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.951313 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.953613 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.954838 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.954986 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.955058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.955523 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.955648 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:03:42.955955 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.948283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:60485 (local address 127.13.162.3:41251)
0924 10:03:42.948474 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:42.948480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:42.949155 (+ 675us) server_negotiation.cc:407] Connection header received
0924 10:03:42.949341 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:42.949345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:42.949412 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:42.949493 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:03:42.950032 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.951112 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.953623 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.954855 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.954990 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.955058 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:42.955523 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 10:03:42.955648 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:03:42.956095 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.948382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:44637 (local address 127.13.162.1:52323)
0924 10:03:42.949251 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.949265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.949383 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.949696 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.949700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.949722 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:03:42.950114 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.950127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.951437 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.951443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.953469 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.953481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.955097 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.955102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.955310 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.955395 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.955491 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.955658 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":69,"thread_start_us":582,"threads_started":1}
I20260924 10:03:42.956205 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:42.948312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41251 (local address 127.13.162.1:60485)
0924 10:03:42.949029 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 10:03:42.949058 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:03:42.949209 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:03:42.949508 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:03:42.949512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:03:42.949535 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:03:42.949895 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.949908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.951230 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.951234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.953467 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:03:42.953482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:03:42.955075 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:03:42.955085 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:03:42.955381 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:42.955408 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:03:42.955491 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:03:42.955658 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":126,"thread_start_us":406,"threads_started":1}
I20260924 10:03:42.957222 25712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d49859e2ef4a79b4e2f12fd7b9139b" candidate_uuid: "a621a2595e42484ab03383c6dfff62ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a843f1d594a84c90be81d68deeeb52f9" is_pre_election: true
I20260924 10:03:42.957221 25578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d49859e2ef4a79b4e2f12fd7b9139b" candidate_uuid: "a621a2595e42484ab03383c6dfff62ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7531814a33444b9db29f4a4239f27c" is_pre_election: true
I20260924 10:03:42.957762 25578 raft_consensus.cc:2454] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a621a2595e42484ab03383c6dfff62ab in term 0.
I20260924 10:03:42.957762 25712 raft_consensus.cc:2454] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a621a2595e42484ab03383c6dfff62ab in term 0.
I20260924 10:03:42.958561 25371 leader_election.cc:302] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c7531814a33444b9db29f4a4239f27c, a621a2595e42484ab03383c6dfff62ab; no voters:
I20260924 10:03:42.959013 25783 raft_consensus.cc:2781] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:03:42.959147 25783 raft_consensus.cc:482] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:03:42.959237 25783 raft_consensus.cc:3037] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:03:42.961359 25783 raft_consensus.cc:504] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.961927 25783 leader_election.cc:288] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [CANDIDATE]: Term 1 election: Requested vote from peers a843f1d594a84c90be81d68deeeb52f9 (127.13.162.3:41251), 3c7531814a33444b9db29f4a4239f27c (127.13.162.2:44637)
I20260924 10:03:42.962446 25578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d49859e2ef4a79b4e2f12fd7b9139b" candidate_uuid: "a621a2595e42484ab03383c6dfff62ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7531814a33444b9db29f4a4239f27c"
I20260924 10:03:42.962481 25712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d49859e2ef4a79b4e2f12fd7b9139b" candidate_uuid: "a621a2595e42484ab03383c6dfff62ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a843f1d594a84c90be81d68deeeb52f9"
I20260924 10:03:42.962653 25712 raft_consensus.cc:3037] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:03:42.962653 25578 raft_consensus.cc:3037] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:03:42.965654 25578 raft_consensus.cc:2454] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a621a2595e42484ab03383c6dfff62ab in term 1.
I20260924 10:03:42.965654 25712 raft_consensus.cc:2454] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a621a2595e42484ab03383c6dfff62ab in term 1.
I20260924 10:03:42.966073 25371 leader_election.cc:302] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c7531814a33444b9db29f4a4239f27c, a621a2595e42484ab03383c6dfff62ab, a843f1d594a84c90be81d68deeeb52f9; no voters:
I20260924 10:03:42.966427 25783 raft_consensus.cc:2781] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:03:42.966898 25783 raft_consensus.cc:686] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 1 LEADER]: Becoming Leader. State: Replica: a621a2595e42484ab03383c6dfff62ab, State: Running, Role: LEADER
I20260924 10:03:42.967641 25783 consensus_queue.cc:237] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } }
I20260924 10:03:42.971845 25267 catalog_manager.cc:5495] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab reported cstate change: term changed from 0 to 1, leader changed from <none> to a621a2595e42484ab03383c6dfff62ab (127.13.162.1). New cstate: current_term: 1 leader_uuid: "a621a2595e42484ab03383c6dfff62ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a621a2595e42484ab03383c6dfff62ab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42675 } health_report { overall_health: HEALTHY } } }
W20260924 10:03:42.987223 25758 tablet.cc:2369] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:03:43.016368 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:43.011224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.13.162.1:42675)
0924 10:03:43.011463 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:43.011467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:43.011482 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:03:43.011568 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:43.011572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:43.011664 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:43.011799 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:03:43.012179 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.012901 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.014253 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.014717 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.014827 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:43.015292 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:03:43.015803 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:03:43.015946 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:43.016038 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 10:03:43.016117 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:03:43.024531 25712 raft_consensus.cc:1261] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 1 FOLLOWER]: Refusing update from remote peer a621a2595e42484ab03383c6dfff62ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:03:43.024549 25578 raft_consensus.cc:1261] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 1 FOLLOWER]: Refusing update from remote peer a621a2595e42484ab03383c6dfff62ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:03:43.025441 25786 consensus_queue.cc:1035] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7531814a33444b9db29f4a4239f27c" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:03:43.025758 25783 consensus_queue.cc:1035] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a843f1d594a84c90be81d68deeeb52f9" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:03:43.053062 25266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:03:43.053438 25266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:03:43.058472 25266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb95129504b243a7ad0d7991ca175d7c: Sending DeleteTablet for 3 replicas of tablet e2d49859e2ef4a79b4e2f12fd7b9139b
I20260924 10:03:43.059950 25692 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d49859e2ef4a79b4e2f12fd7b9139b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:03:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:37420
I20260924 10:03:43.060570 25558 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d49859e2ef4a79b4e2f12fd7b9139b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:03:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:49406
I20260924 10:03:43.060879 25415 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d49859e2ef4a79b4e2f12fd7b9139b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:03:43 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:55214
I20260924 10:03:43.062381 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 25347
I20260924 10:03:43.070225 25799 tablet_replica.cc:330] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: stopping tablet replica
I20260924 10:03:43.070837 25799 raft_consensus.cc:2229] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:03:43.071403 25799 raft_consensus.cc:2258] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:03:43.073325 25801 tablet_replica.cc:330] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: stopping tablet replica
I20260924 10:03:43.073567 25800 tablet_replica.cc:330] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: stopping tablet replica
I20260924 10:03:43.073822 25801 raft_consensus.cc:2229] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:03:43.074054 25800 raft_consensus.cc:2229] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:03:43.074514 25801 raft_consensus.cc:2258] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:03:43.074602 25800 raft_consensus.cc:2258] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:03:43.076689 25801 ts_tablet_manager.cc:1905] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:03:43.076719 25800 ts_tablet_manager.cc:1905] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:03:43.078758 25799 ts_tablet_manager.cc:1905] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:03:43.083598 25800 ts_tablet_manager.cc:1918] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:03:43.083868 25800 log.cc:1196] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/e2d49859e2ef4a79b4e2f12fd7b9139b
I20260924 10:03:43.084152 25801 ts_tablet_manager.cc:1918] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:03:43.084452 25800 ts_tablet_manager.cc:1939] T e2d49859e2ef4a79b4e2f12fd7b9139b P 3c7531814a33444b9db29f4a4239f27c: Deleting consensus metadata
I20260924 10:03:43.084491 25801 log.cc:1196] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/e2d49859e2ef4a79b4e2f12fd7b9139b
I20260924 10:03:43.084986 25801 ts_tablet_manager.cc:1939] T e2d49859e2ef4a79b4e2f12fd7b9139b P a621a2595e42484ab03383c6dfff62ab: Deleting consensus metadata
I20260924 10:03:43.086449 25254 catalog_manager.cc:4841] TS 3c7531814a33444b9db29f4a4239f27c (127.13.162.2:44637): tablet e2d49859e2ef4a79b4e2f12fd7b9139b (table test-table [id=2bc7c14d84f44f45bca1c0526c34f9e8]) successfully deleted
I20260924 10:03:43.086615 25252 catalog_manager.cc:4841] TS a621a2595e42484ab03383c6dfff62ab (127.13.162.1:42675): tablet e2d49859e2ef4a79b4e2f12fd7b9139b (table test-table [id=2bc7c14d84f44f45bca1c0526c34f9e8]) successfully deleted
I20260924 10:03:43.087635 25799 ts_tablet_manager.cc:1918] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:03:43.087791 25799 log.cc:1196] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/e2d49859e2ef4a79b4e2f12fd7b9139b
I20260924 10:03:43.088261 25799 ts_tablet_manager.cc:1939] T e2d49859e2ef4a79b4e2f12fd7b9139b P a843f1d594a84c90be81d68deeeb52f9: Deleting consensus metadata
I20260924 10:03:43.089957 25251 catalog_manager.cc:4841] TS a843f1d594a84c90be81d68deeeb52f9 (127.13.162.3:41251): tablet e2d49859e2ef4a79b4e2f12fd7b9139b (table test-table [id=2bc7c14d84f44f45bca1c0526c34f9e8]) successfully deleted
Sep 24 10:03:43 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:03:43.097741 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:43.063482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.13.162.1:42675)
0924 10:03:43.064603 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:43.064609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:43.072565 (+ 7956us) server_negotiation.cc:407] Connection header received
0924 10:03:43.072809 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:43.072816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:43.072955 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:43.073079 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:43.075309 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.076524 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.086932 (+ 10408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.087386 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.087516 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:43.091608 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:43.091620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:43.091635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:43.091682 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:43.094742 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:43.095445 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:43.095452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:43.095455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:43.095612 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:43.096037 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:43.096041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:43.096043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:43.096420 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:43.096701 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:43.097062 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 10:03:43.097389 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993}
I20260924 10:03:43.098498 25475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:03:43.193605 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 25347
I20260924 10:03:43.215063 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 25486
Sep 24 10:03:43 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:03:43.230558 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:43.215764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.13.162.2:44637)
0924 10:03:43.215935 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:43.215939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:43.216010 (+ 71us) server_negotiation.cc:407] Connection header received
0924 10:03:43.216166 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:43.216176 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:43.216236 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:43.216321 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:43.217861 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.218718 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.220562 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.220979 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.221076 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:43.225185 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:43.225196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:43.225210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:43.225243 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:43.228009 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:43.228721 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:43.228728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:43.228730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:43.228850 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:43.229210 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:43.229212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:43.229214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:43.229561 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:43.229760 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:43.230083 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 10:03:43.230236 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:03:43.231160 25618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:03:43.320855 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 25486
I20260924 10:03:43.333575 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 25629
Sep 24 10:03:43 dist-test-slave-jvsj krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:03:43.348057 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:03:43.334294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.13.162.3:41251)
0924 10:03:43.334494 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:03:43.334500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:03:43.334522 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:03:43.334714 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:03:43.334717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:03:43.334787 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:03:43.334865 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:03:43.336120 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.336824 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.338072 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:03:43.338584 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:03:43.338694 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:03:43.342878 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:03:43.342894 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:03:43.342910 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:03:43.342946 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:03:43.345548 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:43.346252 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:43.346259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:43.346261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:43.346437 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:03:43.346770 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:03:43.346772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:03:43.346774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:03:43.347127 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:03:43.347322 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:03:43.347629 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 10:03:43.347784 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 10:03:43.348699 25752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:03:43.436342 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 25629
I20260924 10:03:43.449185 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 25235
I20260924 10:03:43.450368 25297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:03:43.568137 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 25235
I20260924 10:03:43.585225 13960 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:03:43.930697 13960 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:03:43.930923 13960 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:03:44.266268 13960 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:03:44.266 UTC [24821] LOG: received smart shutdown request
2026-09-24 10:03:44.270 UTC [24821] LOG: background worker "logical replication launcher" (PID 24830) exited with exit code 1
2026-09-24 10:03:44.270 UTC [24825] LOG: shutting down
2026-09-24 10:03:44.419 UTC [24821] LOG: database system is shut down
2026-09-24T10:03:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46500 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:03:44 dist-test-slave-jvsj krb5kdc[25815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:03:44 dist-test-slave-jvsj krb5kdc[25815](info): set up 2 sockets
Sep 24 10:03:44 dist-test-slave-jvsj krb5kdc[25815](info): commencing operation
krb5kdc: starting...
W20260924 10:03:45.530573 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:03:45 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:03:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/rangeradmin_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/rangeradmin_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/rangerlookup_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/rangerlookup_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
I20260924 10:03:45.828819 13960 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYd9YqT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/postgres -l logfile start
2026-09-24 10:03:48.220 UTC [25845] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:03:48.220 UTC [25845] LOG: listening on IPv4 address "127.13.162.20", port 50415
2026-09-24 10:03:48.224 UTC [25845] LOG: listening on Unix socket "/tmp/.s.PGSQL.50415"
2026-09-24 10:03:48.243 UTC [25848] LOG: database system was shut down at 2026-09-24 10:03:46 UTC
2026-09-24 10:03:48.247 UTC [25845] LOG: database system is ready to accept connections
I20260924 10:03:49.142465 13960 mini_postgres.cc:96] Postgres bound to 50415
2026-09-24 10:03:49.161 UTC [25856] FATAL: database "slave" does not exist
127.13.162.20:50415 - accepting connections
I20260924 10:03:49.162292 13960 mini_ranger.cc:161] Starting Ranger...
I20260924 10:03:49.186415 13960 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:03:49.635579 13960 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:03:49.635679 13960 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-admin
2026-09-24 10:03:49,920 [I] DB FLAVOR :POSTGRES
2026-09-24 10:03:49,920 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:03:49,920 [I] Checking connection..
2026-09-24 10:03:49,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:03:50,154 [I] Checking connection passed.
2026-09-24 10:03:50,155 [I] --------- Verifying version history table ---------
2026-09-24 10:03:50,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:03:50,395 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:03:50,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:03:50,645 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:03:50,646 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:03:50,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:03:50.883 UTC [25920] WARNING: there is no transaction in progress
2026-09-24 10:03:50,888 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:03:50,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:03:51,124 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:03:51,124 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:03:51,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:03:51,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:03:51,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jvsj.c.gcp-upstream.internal','N') ;"
2026-09-24 10:03:51,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:03:52,097 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:03:52,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:03:52,332 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:03:52,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:03:52,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:03:52,821 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:03:52,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:03:53.773 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.789 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.807 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.823 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.842 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.858 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.911 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.918 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.932 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.942 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.952 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.966 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.976 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.982 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.988 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:53.995 UTC [26028] WARNING: there is no transaction in progress
2026-09-24 10:03:54,600 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:03:54,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:03:54,842 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:03:54,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:03:55,099 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:03:55,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:03:55,349 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:03:55,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:03:55,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:03:55,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jvsj.c.gcp-upstream.internal';"
2026-09-24 10:03:56,076 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:03:56,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:03:56,338 [I] DB_PATCHES have already been applied
I20260924 10:03:56.343869 13960 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYd9YqT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:03:56.343976 13960 mini_ranger.cc:193] Using host: 127.13.162.20
I20260924 10:03:56.357123 13960 mini_ranger.cc:227] Ranger admin URL: http://127.13.162.20:49713
Sep 24, 2026 10:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:03:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:03:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:03:57.387285 13960 mini_ranger.cc:160] Time spent starting Ranger: real 8.225s user 0.004s sys 0.033s
Sep 24, 2026 10:04:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:04:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 10:04:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:04:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:04:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:04:17.671228 13960 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.162.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.162.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/rangerkms_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.162.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/rangerkms_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.162.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.162.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.162.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/HTTP_127.13.162.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:04:17 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244257, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:04:17.818913 13960 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:04:17.838567 13960 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:04:18.636590 13960 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:04:18.636677 13960 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms
I20260924 10:04:18.636720 13960 mini_ranger_kms.cc:227] Using postgres at 127.13.162.20:50415
2026-09-24 10:04:18,955 [I] DB FLAVOR :POSTGRES
2026-09-24 10:04:18,955 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:04:18,955 [I] Checking connection
2026-09-24 10:04:18,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:04:19,181 [I] connection success
2026-09-24 10:04:19,181 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:04:19,181 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:04:19,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:04:19,406 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:04:19,407 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:04:19,407 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:04:19,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.162.20:50415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:04:19,664 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:04:20.124444 13960 mini_ranger_kms.cc:310] Created kms service
I20260924 10:04:20.303320 13960 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:04:20.348089 13960 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:04:20 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244257, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:20.773366 13960 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:04:20.773654 13960 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYd9YqT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYd9YqT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:04:20.773701 13960 mini_ranger_kms.cc:240] Using host: 127.13.162.20
I20260924 10:04:20.786767 13960 mini_ranger_kms.cc:276] Ranger KMS PID: 26208
I20260924 10:04:20.786893 13960 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.162.20:42271
Sep 24, 2026 10:04:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:04:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:04:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:04:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:04:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:04:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:04:22.829370 13960 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.010s user 0.065s sys 0.265s
I20260924 10:04:22.829587 13960 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:04:22.829739 13960 mini_ranger_kms.cc:401] 127.13.162.20:42271/kms/v1/keys
Sep 24 10:04:22 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244262, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.162.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:04:23 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244262, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.162.20@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
Sep 24 10:04:23 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244262, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.162.20@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
Sep 24 10:04:26 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:04:26.123925 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35687
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35687
--ranger_config_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:26.443313 26259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:26.443620 26259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:26.443740 26259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:26.451182 26259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:26.451301 26259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:26.451364 26259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:26.451440 26259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:26.451502 26259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:26.462601 26259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.162.20:42271/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35687
--ranger_config_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:26.464365 26259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:26.466233 26259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:26.480191 26265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:26.480255 26267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:26.480839 26259 server_base.cc:1034] running on GCE node
W20260924 10:04:26.482568 26264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:26.483436 26259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:26.485021 26259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:26.486251 26259 hybrid_clock.cc:648] HybridClock initialized: now 1790244266486187 us; error 57 us; skew 500 ppm
I20260924 10:04:26.486670 26259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:26 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:26.490095 26259 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:26.491704 26259 webserver.cc:460] Webserver started at http://127.13.162.62:40217/ using document root <none> and password file <none>
I20260924 10:04:26.492372 26259 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:26.492489 26259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:26.492780 26259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:04:26 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:26.544584 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "e387e67b69e444f0b8461d9ddca87b4c"
format_stamp: "Formatted at 2026-09-24 10:04:26 on dist-test-slave-jvsj"
server_key: "1c38a9ad77216da8f41c77034cae32cc"
server_key_iv: "461d7c5236fe0dcd5104a542585d455e"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:26.545737 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "e387e67b69e444f0b8461d9ddca87b4c"
format_stamp: "Formatted at 2026-09-24 10:04:26 on dist-test-slave-jvsj"
server_key: "1c38a9ad77216da8f41c77034cae32cc"
server_key_iv: "461d7c5236fe0dcd5104a542585d455e"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:26.553110 26259 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 10:04:26.570828 26274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:26.572346 26259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:04:26.572526 26259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "e387e67b69e444f0b8461d9ddca87b4c"
format_stamp: "Formatted at 2026-09-24 10:04:26 on dist-test-slave-jvsj"
server_key: "1c38a9ad77216da8f41c77034cae32cc"
server_key_iv: "461d7c5236fe0dcd5104a542585d455e"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:26.572743 26259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:26.600682 26259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:26.604790 26259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:26.605089 26259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:26.622553 26259 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:35687
I20260924 10:04:26.622562 26325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:35687 every 8 connection(s)
I20260924 10:04:26.623912 26259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:26.626456 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 26259
I20260924 10:04:26.626643 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:04:26.627285 13960 external_mini_cluster.cc:1428] Setting key 361283875d0b4782de365d29668418e6
I20260924 10:04:26.627921 26326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:26.640705 26326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Bootstrap starting.
Sep 24 10:04:26 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:26.644635 26326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:26.645731 26326 log.cc:824] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:26.649132 26326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: No bootstrap required, opened a new log
I20260924 10:04:26.649762 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:26.629242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.13.162.62:35687)
0924 10:04:26.630063 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:26.630076 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:26.630119 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:04:26.632642 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:26.632666 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:26.633099 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:26.633513 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:26.634885 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:26.636172 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:26.637466 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:26.638192 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:26.638417 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:26.642636 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:26.642645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:26.642664 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:26.642706 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:26.645683 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:26.646591 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:26.646600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:26.646610 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:26.646828 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:26.647186 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:26.647190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:26.647192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:26.647716 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:26.647924 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:26.648374 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 10:04:26.648698 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":398,"threads_started":1}
I20260924 10:04:26.656562 26326 raft_consensus.cc:348] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:26.656926 26326 raft_consensus.cc:374] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:26.657023 26326 raft_consensus.cc:729] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e387e67b69e444f0b8461d9ddca87b4c, State: Initialized, Role: FOLLOWER
I20260924 10:04:26.657618 26326 consensus_queue.cc:260] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:26.657835 26326 raft_consensus.cc:388] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:04:26.657963 26326 raft_consensus.cc:482] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:04:26.658113 26326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:26.660249 26326 raft_consensus.cc:504] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:26.660801 26326 leader_election.cc:302] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e387e67b69e444f0b8461d9ddca87b4c; no voters:
I20260924 10:04:26.661350 26326 leader_election.cc:288] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:04:26.661463 26331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:04:26.662103 26331 raft_consensus.cc:686] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 LEADER]: Becoming Leader. State: Replica: e387e67b69e444f0b8461d9ddca87b4c, State: Running, Role: LEADER
I20260924 10:04:26.662607 26331 consensus_queue.cc:237] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:26.663333 26326 sys_catalog.cc:564] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:26.665205 26333 sys_catalog.cc:455] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e387e67b69e444f0b8461d9ddca87b4c. Latest consensus state: current_term: 1 leader_uuid: "e387e67b69e444f0b8461d9ddca87b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } } }
I20260924 10:04:26.665222 26332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e387e67b69e444f0b8461d9ddca87b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } } }
I20260924 10:04:26.665640 26333 sys_catalog.cc:458] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:26.665648 26332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:26.666503 26336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:26.672879 26336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:26.682238 26336 catalog_manager.cc:1347] Generated new cluster ID: efebb54d0dab42629fc23bb21b85b875
I20260924 10:04:26.682396 26336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:04:26.688351 26326 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:04:26.708379 26336 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:04:26.710502 26336 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:04:26.729908 26336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Generated new TSK 0
I20260924 10:04:26.731178 26336 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:04:27.471216 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35687
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:04:27 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:04:27.873637 26371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:27.873976 26371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:27.874107 26371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:27.881587 26371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:27.881711 26371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:27.881887 26371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:04:27.892527 26371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.162.20:42271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:27.893870 26371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:27.895237 26371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:27.910633 26381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:27.910794 26382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:27.911783 26384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:27.913120 26371 server_base.cc:1034] running on GCE node
I20260924 10:04:27.913800 26371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:27.916585 26371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:27.917843 26371 hybrid_clock.cc:648] HybridClock initialized: now 1790244267917773 us; error 55 us; skew 500 ppm
I20260924 10:04:27.918278 26371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:27 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:27.924650 26371 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:04:27.926468 26371 webserver.cc:460] Webserver started at http://127.13.162.1:42229/ using document root <none> and password file <none>
I20260924 10:04:27.927188 26371 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:27.927300 26371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:27.927600 26371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:04:27 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:27.948122 26371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "bb9e1bd39ac94b92a25899eb8fa2889c"
format_stamp: "Formatted at 2026-09-24 10:04:27 on dist-test-slave-jvsj"
server_key: "7ac59d09e8738732a2be989f8f75c2b5"
server_key_iv: "eaf891cf8d7667ef815007f861c99e49"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:27.949290 26371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "bb9e1bd39ac94b92a25899eb8fa2889c"
format_stamp: "Formatted at 2026-09-24 10:04:27 on dist-test-slave-jvsj"
server_key: "7ac59d09e8738732a2be989f8f75c2b5"
server_key_iv: "eaf891cf8d7667ef815007f861c99e49"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:27.962394 26371 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.006s
I20260924 10:04:27.984114 26391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:27.986189 26371 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260924 10:04:27.986496 26371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "bb9e1bd39ac94b92a25899eb8fa2889c"
format_stamp: "Formatted at 2026-09-24 10:04:27 on dist-test-slave-jvsj"
server_key: "7ac59d09e8738732a2be989f8f75c2b5"
server_key_iv: "eaf891cf8d7667ef815007f861c99e49"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:27.986825 26371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:28.045559 26371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:28.051193 26371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:28.051621 26371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:28.064250 26371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:28.067278 26371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:28.067461 26371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 10:04:28.067657 26371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:28.067765 26371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:04:28 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:28.564080 26371 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:44857
I20260924 10:04:28.564133 26503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:44857 every 8 connection(s)
I20260924 10:04:28.565826 26371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:04:28.574097 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 26371
I20260924 10:04:28.574352 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:04:28.575177 13960 external_mini_cluster.cc:1428] Setting key 50efb723c259ad188894b2b5a55fe89f
Sep 24 10:04:28 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:04:28.628727 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:28.570504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:46015 (local address 127.13.162.62:35687)
0924 10:04:28.606182 (+ 35678us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:28.606188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:28.606215 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:04:28.607867 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:28.607874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:28.607984 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:28.608069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:28.611124 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:28.612345 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:28.614879 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:28.615504 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:28.615617 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:28.620749 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:28.620757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:28.620763 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:28.620820 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:28.624668 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:28.625778 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:28.625784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:28.625788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:28.625958 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:28.626567 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:28.626572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:28.626576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:28.626936 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:28.627065 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:28.628075 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0924 10:04:28.628322 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35526,"thread_start_us":35691,"threads_started":1}
I20260924 10:04:28.631080 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:28.569650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35687 (local address 127.13.162.1:46015)
0924 10:04:28.603027 (+ 33377us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:28.603100 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:28.607481 (+ 4381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:28.608668 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:28.608683 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:28.609377 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:28.610283 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:28.610310 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:28.612571 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:28.612576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:28.614667 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:28.614683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:28.615958 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:28.615965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:28.616265 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:28.617056 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:28.617086 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:28.620386 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:28.624923 (+ 4537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:28.624934 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:28.624956 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:28.625620 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:28.626115 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:28.626120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:28.626124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:28.626420 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:28.627160 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:28.627169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:28.627653 (+ 484us) client_negotiation.cc:769] Sending connection context
0924 10:04:28.629278 (+ 1625us) client_negotiation.cc:240] Negotiation successful
0924 10:04:28.629876 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32977,"thread_start_us":35107,"threads_started":1}
I20260924 10:04:28.632853 26504 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35687
I20260924 10:04:28.633230 26504 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:04:28.634095 26504 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:28.640089 26291 ts_manager.cc:194] Registered new tserver with Master: bb9e1bd39ac94b92a25899eb8fa2889c (127.13.162.1:44857)
I20260924 10:04:28.641590 26291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:46015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:04:28.663511 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35687
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:04:28 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
W20260924 10:04:29.080397 26510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:29.080749 26510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:29.080873 26510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:29.088410 26510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:29.088541 26510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:29.088683 26510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:04:29.099035 26510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.162.20:42271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:29.100400 26510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:29.101688 26510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:29.116572 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:29.116549 26525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:29.117146 26527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:29.117811 26510 server_base.cc:1034] running on GCE node
I20260924 10:04:29.118621 26510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:29.119807 26510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:29.121078 26510 hybrid_clock.cc:648] HybridClock initialized: now 1790244269121010 us; error 50 us; skew 500 ppm
I20260924 10:04:29.121449 26510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:29.125198 26510 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:04:29.126750 26510 webserver.cc:460] Webserver started at http://127.13.162.2:38081/ using document root <none> and password file <none>
I20260924 10:04:29.127421 26510 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:29.127533 26510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:29.127839 26510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:29.145578 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "a88a0d9a238f488d88e9b120e54bdc70"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "3d2f0dc8f1effe35290d59c7a529ebe2"
server_key_iv: "65329e11237ad4c764002e1b632528d3"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:29.147063 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "a88a0d9a238f488d88e9b120e54bdc70"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "3d2f0dc8f1effe35290d59c7a529ebe2"
server_key_iv: "65329e11237ad4c764002e1b632528d3"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:29.154358 26510 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 10:04:29.172863 26534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:29.174850 26510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 10:04:29.175127 26510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "a88a0d9a238f488d88e9b120e54bdc70"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "3d2f0dc8f1effe35290d59c7a529ebe2"
server_key_iv: "65329e11237ad4c764002e1b632528d3"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:29.175472 26510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:29.201135 26510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:29.206488 26510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:29.206864 26510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:29.208168 26510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:29.210000 26510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:29.210160 26510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:29.210340 26510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:29.210433 26510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:29.254540 26510 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:37335
I20260924 10:04:29.254592 26646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:37335 every 8 connection(s)
I20260924 10:04:29.256219 26510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:04:29.257515 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 26510
I20260924 10:04:29.257691 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:04:29.258446 13960 external_mini_cluster.cc:1428] Setting key 170527e2dbc5d41f032773ed8f03c1c8
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:04:29.296505 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:29.263078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:46823 (local address 127.13.162.62:35687)
0924 10:04:29.274826 (+ 11748us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:29.274834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:29.276134 (+ 1300us) server_negotiation.cc:407] Connection header received
0924 10:04:29.278195 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:29.278200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:29.278288 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:29.278409 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:29.280618 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.281708 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.284043 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.284610 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.284706 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:29.288998 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:29.289006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:29.289011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:29.289051 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:29.292563 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.293776 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.293783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.293787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.293944 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.294530 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.294534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.294537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.294975 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:29.295119 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:29.295878 (+ 759us) server_negotiation.cc:299] Negotiation successful
0924 10:04:29.296130 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11607,"thread_start_us":11498,"threads_started":1}
I20260924 10:04:29.297580 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:29.262462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35687 (local address 127.13.162.2:46823)
0924 10:04:29.275449 (+ 12987us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:29.275506 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:29.277707 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:29.278896 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:29.278908 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:29.279501 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:29.280403 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:29.280430 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.281882 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.281887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:29.283868 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:29.283884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.285040 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.285046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:29.285333 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:29.286079 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:29.286108 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:29.288670 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:29.292765 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:29.292776 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:29.292797 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:29.293605 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:29.294087 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:29.294092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:29.294095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:29.294334 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:29.295130 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:29.295139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:29.295510 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 10:04:29.295751 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 10:04:29.296290 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12669,"thread_start_us":12429,"threads_started":1}
I20260924 10:04:29.299192 26647 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35687
I20260924 10:04:29.299582 26647 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
I20260924 10:04:29.300482 26647 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:29.302206 26291 ts_manager.cc:194] Registered new tserver with Master: a88a0d9a238f488d88e9b120e54bdc70 (127.13.162.2:37335)
I20260924 10:04:29.303265 26291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:46823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:04:29.327636 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35687
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:04:29.644876 26504 heartbeater.cc:503] Master 127.13.162.62:35687 was elected leader, sending a full tablet report...
W20260924 10:04:29.659189 26653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:29.659493 26653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:29.659631 26653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:29.666961 26653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:29.667094 26653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:29.667240 26653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:04:29.677246 26653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.162.20:42271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:29.678623 26653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:29.680008 26653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:29.692389 26661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:29.692714 26659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:29.692451 26658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:29.693380 26653 server_base.cc:1034] running on GCE node
I20260924 10:04:29.693976 26653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:29.694975 26653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:29.696173 26653 hybrid_clock.cc:648] HybridClock initialized: now 1790244269696105 us; error 63 us; skew 500 ppm
I20260924 10:04:29.696512 26653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:29.699436 26653 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:04:29.700891 26653 webserver.cc:460] Webserver started at http://127.13.162.3:39843/ using document root <none> and password file <none>
I20260924 10:04:29.701546 26653 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:29.701664 26653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:29.701949 26653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:29.716437 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "d9613ae4e7324d40a40ef44b304d2a47"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "e96c5f086988cd335d8b36369613b148"
server_key_iv: "6ae1fc4a18289576103d21b3559d21b6"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:29.717341 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "d9613ae4e7324d40a40ef44b304d2a47"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "e96c5f086988cd335d8b36369613b148"
server_key_iv: "6ae1fc4a18289576103d21b3559d21b6"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:29.722817 26653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:04:29.736709 26668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:29.738528 26653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:04:29.738709 26653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "d9613ae4e7324d40a40ef44b304d2a47"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "e96c5f086988cd335d8b36369613b148"
server_key_iv: "6ae1fc4a18289576103d21b3559d21b6"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:29.738922 26653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:29.755790 26653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:29.760314 26653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:29.760620 26653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:29.761695 26653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:29.763187 26653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:29.763273 26653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:29.763392 26653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:29.763449 26653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:29.797115 26653 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:45915
I20260924 10:04:29.797143 26780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:45915 every 8 connection(s)
I20260924 10:04:29.798642 26653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:04:29.798972 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 26653
I20260924 10:04:29.799132 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:04:29.799718 13960 external_mini_cluster.cc:1428] Setting key c346752243a2e71977a11c1cbc399b62
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:29.820044 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:29.801145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:60031 (local address 127.13.162.62:35687)
0924 10:04:29.806757 (+ 5612us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:29.806761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:29.806777 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:04:29.806846 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:29.806849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:29.806921 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:29.807024 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:29.808520 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.809211 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.810826 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.811221 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.811309 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:29.814611 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:29.814616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:29.814619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:29.814650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:29.817320 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.818014 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.818019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.818021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.818163 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.818569 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.818573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.818575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.818852 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:29.818963 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:29.819615 (+ 652us) server_negotiation.cc:299] Negotiation successful
0924 10:04:29.819789 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5515,"thread_start_us":5358,"threads_started":1}
I20260924 10:04:29.820962 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:29.801771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35687 (local address 127.13.162.3:60031)
0924 10:04:29.802605 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:29.802658 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:29.804320 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:29.807325 (+ 3005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:29.807335 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:29.807773 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:29.808380 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:29.808397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.809348 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.809352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:29.810693 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:29.810705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.811404 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.811409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:29.811618 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:29.812218 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:29.812240 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:29.814388 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:29.817437 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:29.817444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:29.817458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:29.817871 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:29.818244 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:29.818247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:29.818249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:29.818483 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:29.818943 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:29.818949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:29.819254 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 10:04:29.819470 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 10:04:29.819895 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":323,"threads_started":1}
I20260924 10:04:29.822265 26781 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35687
I20260924 10:04:29.822592 26781 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:29.823318 26781 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:29.824682 26291 ts_manager.cc:194] Registered new tserver with Master: d9613ae4e7324d40a40ef44b304d2a47 (127.13.162.3:45915)
I20260924 10:04:29.825464 26291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:60031
I20260924 10:04:29.828161 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:04:29.829735 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 26371
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:04:29.845803 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:29.830649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.13.162.1:44857)
0924 10:04:29.831217 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:29.831224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:29.831248 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:04:29.831301 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:29.831308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:29.831505 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:29.831631 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:29.832995 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.834006 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.835474 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.836040 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.836139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:29.840310 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:29.840324 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:29.840340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:29.840375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:29.843193 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.843902 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.843910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.843912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.844092 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.844382 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.844385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.844387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.844759 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:29.845013 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:29.845244 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 10:04:29.845445 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260924 10:04:29.847126 26499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:29.929898 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 26371
I20260924 10:04:29.942147 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 26510
Sep 24 10:04:29 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:04:29.958300 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:29.943157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.13.162.2:37335)
0924 10:04:29.943738 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:29.943746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:29.943773 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:04:29.943833 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:29.943839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:29.944078 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:29.944237 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:29.945452 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.946512 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.947941 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:29.948498 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:29.948624 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:29.952891 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:29.952904 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:29.952919 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:29.952956 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:29.955505 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.956229 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.956236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.956238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.956368 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:29.956714 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:29.956718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:29.956721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:29.957077 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:29.957373 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:29.957677 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 10:04:29.957941 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":281,"threads_started":1}
I20260924 10:04:29.959699 26642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:30.040964 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 26510
I20260924 10:04:30.053325 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 26653
Sep 24 10:04:30 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:04:30.071137 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:30.054622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.13.162.3:45915)
0924 10:04:30.055322 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:30.055329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:30.055360 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:04:30.055424 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:30.055431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:30.055723 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:30.055872 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:30.057260 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:30.058411 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:30.059907 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:30.060547 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:30.060652 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:30.065254 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:30.065268 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:30.065285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:30.065329 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:30.068254 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:30.069042 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:30.069050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:30.069053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:30.069191 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:30.069585 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:30.069588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:30.069590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:30.069943 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:30.070185 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:30.070576 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 10:04:30.070785 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":407,"threads_started":1}
I20260924 10:04:30.072518 26776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:30.154042 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 26653
I20260924 10:04:30.167310 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 26259
I20260924 10:04:30.168661 26321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:30.274443 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 26259
I20260924 10:04:30.292220 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35687
--webserver_interface=127.13.162.62
--webserver_port=40217
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35687
--ranger_config_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:30.624375 26793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:30.624692 26793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:30.624784 26793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:30.632232 26793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:30.632367 26793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:30.632437 26793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:30.632516 26793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:30.632572 26793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:30.643486 26793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35687
--ranger_config_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=40217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:30.644796 26793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:30.646103 26793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:30.658036 26798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:30.658125 26801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:30.658105 26799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:30.658895 26793 server_base.cc:1034] running on GCE node
I20260924 10:04:30.659703 26793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:30.661171 26793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:30.662417 26793 hybrid_clock.cc:648] HybridClock initialized: now 1790244270662359 us; error 52 us; skew 500 ppm
I20260924 10:04:30.662814 26793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:30 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244270, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:30.666103 26793 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:30.667752 26793 webserver.cc:460] Webserver started at http://127.13.162.62:40217/ using document root <none> and password file <none>
I20260924 10:04:30.668466 26793 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:30.668594 26793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:04:30 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244270, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:30.728581 26793 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:04:30.731833 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:30.733559 26793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:04:30.733758 26793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "e387e67b69e444f0b8461d9ddca87b4c"
format_stamp: "Formatted at 2026-09-24 10:04:26 on dist-test-slave-jvsj"
server_key: "1c38a9ad77216da8f41c77034cae32cc"
server_key_iv: "461d7c5236fe0dcd5104a542585d455e"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:30.734809 26793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:30.773159 26793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:30.777381 26793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:30.777663 26793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:30.796082 26793 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:35687
I20260924 10:04:30.796100 26860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:35687 every 8 connection(s)
I20260924 10:04:30.797605 26793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:30.804757 26861 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:04:30.806155 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 26793
I20260924 10:04:30.807229 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:44857
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:35687
--webserver_port=42229
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:04:30.809181 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Bootstrap starting.
I20260924 10:04:30.829608 26861 log.cc:824] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:30.839286 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:04:30.839848 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Bootstrap complete.
I20260924 10:04:30.848836 26861 raft_consensus.cc:348] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:30.849707 26861 raft_consensus.cc:729] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e387e67b69e444f0b8461d9ddca87b4c, State: Initialized, Role: FOLLOWER
I20260924 10:04:30.850370 26861 consensus_queue.cc:260] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:30.850600 26861 raft_consensus.cc:388] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:04:30.850718 26861 raft_consensus.cc:482] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:04:30.850879 26861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:04:30.855545 26861 raft_consensus.cc:504] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:30.856070 26861 leader_election.cc:302] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e387e67b69e444f0b8461d9ddca87b4c; no voters:
I20260924 10:04:30.856662 26861 leader_election.cc:288] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:04:30.856801 26865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:04:30.857904 26865 raft_consensus.cc:686] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [term 2 LEADER]: Becoming Leader. State: Replica: e387e67b69e444f0b8461d9ddca87b4c, State: Running, Role: LEADER
I20260924 10:04:30.858608 26865 consensus_queue.cc:237] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } }
I20260924 10:04:30.858991 26861 sys_catalog.cc:564] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:30.861117 26867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e387e67b69e444f0b8461d9ddca87b4c. Latest consensus state: current_term: 2 leader_uuid: "e387e67b69e444f0b8461d9ddca87b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } } }
I20260924 10:04:30.861192 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e387e67b69e444f0b8461d9ddca87b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e387e67b69e444f0b8461d9ddca87b4c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35687 } } }
I20260924 10:04:30.861480 26867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:30.861532 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:30.862383 26870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:30.875640 26870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:30.877290 26870 catalog_manager.cc:1259] Loaded cluster ID: efebb54d0dab42629fc23bb21b85b875
I20260924 10:04:30.877457 26870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:04:30.881636 26870 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:04:30.883206 26870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Loaded TSK: 0
I20260924 10:04:30.885072 26870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:04:30.897691 26861 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 10:04:31.240914 26863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:31.241425 26863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:31.241617 26863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:31.253268 26863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:31.253509 26863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:31.253742 26863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:04:31.269510 26863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:44857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=42229
--tserver_master_addrs=127.13.162.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:31.271669 26863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:31.273542 26863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:31.302681 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:31.302876 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:31.314206 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:31.316642 26863 server_base.cc:1034] running on GCE node
I20260924 10:04:31.317653 26863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:31.319703 26863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:31.321152 26863 hybrid_clock.cc:648] HybridClock initialized: now 1790244271321026 us; error 107 us; skew 500 ppm
I20260924 10:04:31.321623 26863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:31 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:31.326143 26863 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:04:31.337445 26863 webserver.cc:460] Webserver started at http://127.13.162.1:42229/ using document root <none> and password file <none>
I20260924 10:04:31.338697 26863 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:31.338891 26863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:04:31 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:31.388118 26863 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260924 10:04:31.395246 26909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:31.397512 26863 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260924 10:04:31.397846 26863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "bb9e1bd39ac94b92a25899eb8fa2889c"
format_stamp: "Formatted at 2026-09-24 10:04:27 on dist-test-slave-jvsj"
server_key: "7ac59d09e8738732a2be989f8f75c2b5"
server_key_iv: "eaf891cf8d7667ef815007f861c99e49"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:31.399371 26863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:31.431442 26863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:31.437139 26863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:31.437464 26863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:31.438995 26863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:31.440829 26863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:31.440938 26863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:31.441063 26863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:31.441160 26863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:31.513898 26863 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:44857
I20260924 10:04:31.513968 27022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:44857 every 8 connection(s)
I20260924 10:04:31.515722 26863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:04:31.524879 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 26863
I20260924 10:04:31.525709 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:37335
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:35687
--webserver_port=38081
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:04:31 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:31.547080 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:31.519156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35687 (local address 127.13.162.1:57049)
0924 10:04:31.520173 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:31.520211 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:31.521980 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:31.523722 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:31.523735 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:31.525418 (+ 1683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:31.526578 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:31.526622 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:31.529047 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:31.529053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:31.531058 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:31.531077 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:31.532899 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:31.532905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:31.533174 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:31.533941 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:31.533973 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:31.536777 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:31.541497 (+ 4720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:31.541508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:31.541526 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:31.542196 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:31.542820 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:31.542825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:31.542828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:31.543087 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:31.544130 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:31.544139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:31.544608 (+ 469us) client_negotiation.cc:769] Sending connection context
0924 10:04:31.544929 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 10:04:31.545526 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":566,"threads_started":1}
I20260924 10:04:31.549942 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:31.518941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:57049 (local address 127.13.162.62:35687)
0924 10:04:31.519673 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:31.519692 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:31.522348 (+ 2656us) server_negotiation.cc:407] Connection header received
0924 10:04:31.522752 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:31.522768 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:31.523150 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:31.523498 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:31.526932 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:31.528817 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:31.531263 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:31.532312 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:31.532490 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:31.537125 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:31.537136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:31.537162 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:31.537220 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:31.541291 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:31.542424 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:31.542434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:31.542443 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:31.542681 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:31.543225 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:31.543230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:31.543233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:31.543945 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:31.544144 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:31.548044 (+ 3900us) server_negotiation.cc:299] Negotiation successful
0924 10:04:31.548510 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":422,"threads_started":1}
I20260924 10:04:31.553820 27023 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35687
I20260924 10:04:31.554165 27023 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:31.555100 27023 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
W20260924 10:04:31.558516 27023 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:35687 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:04:31.558770 27023 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:31.559079 27023 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:31.560366 27023 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:31.560719 27023 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
W20260924 10:04:31.561820 27023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:04:31.925276 27027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:31.925585 27027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:31.925721 27027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:31.933105 27027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:31.933234 27027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:31.933420 27027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:04:31.943785 27027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:37335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=38081
--tserver_master_addrs=127.13.162.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:31.945484 27027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:31.947124 27027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:31.962311 27041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:31.962682 27042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:31.968118 27027 server_base.cc:1034] running on GCE node
W20260924 10:04:31.968171 27044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:31.968925 27027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:31.969947 27027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:31.971159 27027 hybrid_clock.cc:648] HybridClock initialized: now 1790244271971122 us; error 41 us; skew 500 ppm
I20260924 10:04:31.971479 27027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:31 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:31.974851 27027 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:04:32.002838 27027 webserver.cc:460] Webserver started at http://127.13.162.2:38081/ using document root <none> and password file <none>
I20260924 10:04:32.004086 27027 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:32.004336 27027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:32.048434 27027 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260924 10:04:32.052752 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:32.054723 27027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:04:32.054980 27027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "a88a0d9a238f488d88e9b120e54bdc70"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "3d2f0dc8f1effe35290d59c7a529ebe2"
server_key_iv: "65329e11237ad4c764002e1b632528d3"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:32.056494 27027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244272, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:32.090677 27027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:32.096407 27027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:32.097142 27027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:32.105252 27027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:32.114055 27027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:32.114267 27027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 10:04:32.114455 27027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:32.114564 27027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:32.167789 27027 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:37335
I20260924 10:04:32.167858 27167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:37335 every 8 connection(s)
I20260924 10:04:32.169787 27027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:04:32.179067 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27027
I20260924 10:04:32.179831 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.162.20:42271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:45915
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:35687
--webserver_port=39843
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:32.206913 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:32.174411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:44385 (local address 127.13.162.62:35687)
0924 10:04:32.178689 (+ 4278us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:32.178696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:32.188994 (+ 10298us) server_negotiation.cc:407] Connection header received
0924 10:04:32.191468 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:32.191474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:32.191591 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:32.191699 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:32.193543 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:32.194491 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:32.196262 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:32.196688 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:32.196801 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:32.200061 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:32.200066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:32.200069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:32.200103 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:32.202749 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:32.203672 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:32.203678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:32.203682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:32.203835 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:32.204460 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:32.204465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:32.204469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:32.204906 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:32.205030 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:32.206206 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0924 10:04:32.206529 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4135,"thread_start_us":4020,"threads_started":1}
I20260924 10:04:32.209389 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:32.173731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35687 (local address 127.13.162.2:44385)
0924 10:04:32.188787 (+ 15056us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:32.188846 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:32.191043 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:32.192173 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:32.192185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:32.192722 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:32.193336 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:32.193356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:32.194685 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:32.194691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:32.196114 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:32.196128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:32.196917 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:32.196924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:32.197162 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:32.197794 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:32.197817 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:32.199838 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:32.202892 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:32.202901 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:32.202920 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:32.203520 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:32.203986 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:32.203991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:32.203995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:32.204322 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:32.205152 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:32.205161 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:32.205736 (+ 575us) client_negotiation.cc:769] Sending connection context
0924 10:04:32.207161 (+ 1425us) client_negotiation.cc:240] Negotiation successful
0924 10:04:32.207748 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14685,"thread_start_us":17629,"threads_started":1}
I20260924 10:04:32.211216 27168 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35687
I20260924 10:04:32.211545 27168 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:32.212404 27168 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:32.214481 26826 ts_manager.cc:194] Registered new tserver with Master: a88a0d9a238f488d88e9b120e54bdc70 (127.13.162.2:37335)
I20260924 10:04:32.216234 26826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:44385
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244272, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:32.561965 27023 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:32.562445 27023 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:32.563725 26826 ts_manager.cc:194] Registered new tserver with Master: bb9e1bd39ac94b92a25899eb8fa2889c (127.13.162.1:44857)
I20260924 10:04:32.564841 26826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:57049
W20260924 10:04:32.668658 27172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:32.669078 27172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:32.669252 27172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:32.681018 27172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:32.681195 27172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:32.681382 27172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:04:32.697211 27172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.162.20:42271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:45915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=39843
--tserver_master_addrs=127.13.162.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:32.698940 27172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:32.700642 27172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:32.746714 27181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:32.747408 27172 server_base.cc:1034] running on GCE node
W20260924 10:04:32.747891 27177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:32.753100 27178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:32.754046 27172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:32.762288 27172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:32.769860 27172 hybrid_clock.cc:648] HybridClock initialized: now 1790244272769759 us; error 94 us; skew 500 ppm
I20260924 10:04:32.770370 27172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244272, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:32.775000 27172 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:04:32.807927 27172 webserver.cc:460] Webserver started at http://127.13.162.3:39843/ using document root <none> and password file <none>
I20260924 10:04:32.808861 27172 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:32.809012 27172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244272, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:32.836509 27172 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260924 10:04:32.840701 27188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:32.842733 27172 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:04:32.842994 27172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "d9613ae4e7324d40a40ef44b304d2a47"
format_stamp: "Formatted at 2026-09-24 10:04:29 on dist-test-slave-jvsj"
server_key: "e96c5f086988cd335d8b36369613b148"
server_key_iv: "6ae1fc4a18289576103d21b3559d21b6"
server_key_version: "kuduclusterkey@0"
I20260924 10:04:32.844506 27172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:32.872607 27172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 24 10:04:32 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244272, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for HTTP/127.13.162.20@KRBTEST.COM
I20260924 10:04:32.877703 27172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:32.878065 27172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:32.879367 27172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:32.881126 27172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:32.881237 27172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:32.881338 27172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:32.881412 27172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:33.130124 27301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:45915 every 8 connection(s)
I20260924 10:04:33.130124 27172 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:45915
I20260924 10:04:33.132091 27172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:04:33.138303 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27172
Sep 24 10:04:33 dist-test-slave-jvsj krb5kdc[25815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244272, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:33.166293 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.134757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:40717 (local address 127.13.162.62:35687)
0924 10:04:33.135163 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.135167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.136374 (+ 1207us) server_negotiation.cc:407] Connection header received
0924 10:04:33.138216 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.138222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.138354 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.138462 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:33.141316 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.142582 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.145213 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.145963 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.146068 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.158178 (+ 12110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:33.158185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:33.158190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:33.158244 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:33.161893 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.162960 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.162966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.162970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.163120 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.163689 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.163694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.163697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.164031 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:33.164193 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.165621 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.165870 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":208,"threads_started":1}
I20260924 10:04:33.167117 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.135473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35687 (local address 127.13.162.3:40717)
0924 10:04:33.136209 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:33.136256 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:33.137919 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:33.139377 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:33.139390 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:33.140140 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:33.141060 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.141087 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.142815 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.142820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.145002 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.145020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.146486 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.146494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.146825 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.154995 (+ 8170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:33.155039 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:33.157859 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:33.162092 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:33.162104 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:33.162126 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:33.162836 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:33.163230 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:33.163236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:33.163239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:33.163550 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:33.164283 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:33.164293 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:33.164756 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 10:04:33.165074 (+ 318us) client_negotiation.cc:240] Negotiation successful
0924 10:04:33.165667 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":295,"threads_started":1}
I20260924 10:04:33.169063 27302 heartbeater.cc:347] Connected to a master server at 127.13.162.62:35687
I20260924 10:04:33.169404 27302 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:33.170270 27302 heartbeater.cc:511] Master 127.13.162.62:35687 requested a full tablet report, sending...
I20260924 10:04:33.171813 26826 ts_manager.cc:194] Registered new tserver with Master: d9613ae4e7324d40a40ef44b304d2a47 (127.13.162.3:45915)
I20260924 10:04:33.172863 26826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:40717
I20260924 10:04:33.176545 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.139334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.13.162.62:35687)
0924 10:04:33.147968 (+ 8634us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.147974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.148685 (+ 711us) server_negotiation.cc:407] Connection header received
0924 10:04:33.148743 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.148748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.148824 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.148919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:33.151404 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.152576 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.156474 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.162209 (+ 5735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.162339 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.165921 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:33.165927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:33.165931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:33.165965 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:33.170955 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.174535 (+ 3580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.174541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.174544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.174695 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.175324 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.175328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.175330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.175581 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:33.175784 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.176054 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.176244 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8483,"thread_start_us":8331,"threads_started":1}
I20260924 10:04:33.178862 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:04:33.192876 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.181635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.13.162.62:35687)
0924 10:04:33.181839 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.181845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.182058 (+ 213us) server_negotiation.cc:407] Connection header received
0924 10:04:33.182178 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.182181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.182247 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.182340 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:33.183689 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.184349 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.185619 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.186022 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.186108 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.188348 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:33.188352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:33.188356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:33.188384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:33.190834 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.191401 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.191405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.191407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.191518 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.191793 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.191796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.191798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.192044 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:33.192217 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.192479 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.192639 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:04:33.197592 26826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:04:33.204689 26826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:04:33.220240 27168 heartbeater.cc:503] Master 127.13.162.62:35687 was elected leader, sending a full tablet report...
I20260924 10:04:33.235745 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.218381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44857 (local address 127.0.0.1:58536)
0924 10:04:33.226224 (+ 7843us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:33.226244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:33.226390 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:33.228313 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:33.228320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:33.228354 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:33.228840 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.228852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.230977 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.230982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.233148 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.233160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.234696 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.234703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.234900 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.234953 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 10:04:33.235084 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 10:04:33.235287 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7711,"mutex_wait_us":117,"thread_start_us":6643,"threads_started":1}
I20260924 10:04:33.236425 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.218353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:45915 (local address 127.0.0.1:55670)
0924 10:04:33.225511 (+ 7158us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:33.225558 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:33.225762 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:33.228591 (+ 2829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:33.228596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:33.228618 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:33.229037 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.229047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.230903 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.230908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.233911 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.233926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.235631 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.235637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.235812 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.235843 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:04:33.235915 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:04:33.236051 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6828,"mutex_wait_us":467,"thread_start_us":7595,"threads_started":1}
I20260924 10:04:33.236680 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.218522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.13.162.1:44857)
0924 10:04:33.227650 (+ 9128us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.227659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.227690 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:04:33.227798 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.227806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.228070 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.229074 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:33.229282 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.230798 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.233307 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.234473 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.234615 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.236082 (+ 1467us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.236172 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.236351 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8985,"thread_start_us":8831,"threads_started":1}
I20260924 10:04:33.236956 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.227171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:37335 (local address 127.0.0.1:47278)
0924 10:04:33.228354 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:33.228369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:33.228567 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:33.229542 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:33.229547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:33.229575 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:33.229911 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.229923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.231752 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.231756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.233911 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.233926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.235956 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.235962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.236146 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.236181 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:04:33.236261 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:04:33.236387 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":5,"thread_start_us":913,"threads_started":1}
I20260924 10:04:33.237000 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.226576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.13.162.2:37335)
0924 10:04:33.227199 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.227209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.228459 (+ 1250us) server_negotiation.cc:407] Connection header received
0924 10:04:33.228748 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.228756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.229064 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.229234 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:33.230185 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.231618 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.234081 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.235455 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.235568 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.235804 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.236521 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.236693 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":255,"threads_started":1}
I20260924 10:04:33.238029 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.219201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.13.162.3:45915)
0924 10:04:33.228023 (+ 8822us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.228031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.228058 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:04:33.228143 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.228150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.228387 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.229153 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:33.229439 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.230769 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.234110 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.235388 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.237267 (+ 1879us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.237482 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.237563 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.237728 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8666,"thread_start_us":8526,"threads_started":1}
I20260924 10:04:33.240440 26958 tablet_service.cc:1467] Processing CreateTablet for tablet 03b75c8cbdef40cfa58eb0277a6522f4 (DEFAULT_TABLE table=test-table [id=fcd40d8e0ec5481aa294ba9aa638755f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:33.240720 27102 tablet_service.cc:1467] Processing CreateTablet for tablet 03b75c8cbdef40cfa58eb0277a6522f4 (DEFAULT_TABLE table=test-table [id=fcd40d8e0ec5481aa294ba9aa638755f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:33.241695 26958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b75c8cbdef40cfa58eb0277a6522f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:33.241329 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 03b75c8cbdef40cfa58eb0277a6522f4 (DEFAULT_TABLE table=test-table [id=fcd40d8e0ec5481aa294ba9aa638755f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:33.242030 27102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b75c8cbdef40cfa58eb0277a6522f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:33.242545 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b75c8cbdef40cfa58eb0277a6522f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:33.254470 27330 tablet_bootstrap.cc:492] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Bootstrap starting.
I20260924 10:04:33.257315 27332 tablet_bootstrap.cc:492] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Bootstrap starting.
I20260924 10:04:33.258060 27331 tablet_bootstrap.cc:492] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Bootstrap starting.
I20260924 10:04:33.258548 27330 tablet_bootstrap.cc:654] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:33.259919 27330 log.cc:824] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:33.261379 27331 tablet_bootstrap.cc:654] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:33.261389 27332 tablet_bootstrap.cc:654] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:33.262596 27332 log.cc:824] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:33.262650 27331 log.cc:824] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:33.263830 27330 tablet_bootstrap.cc:492] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: No bootstrap required, opened a new log
I20260924 10:04:33.264083 27330 ts_tablet_manager.cc:1397] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 10:04:33.265188 27332 tablet_bootstrap.cc:492] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: No bootstrap required, opened a new log
I20260924 10:04:33.265421 27332 ts_tablet_manager.cc:1397] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260924 10:04:33.267470 27331 tablet_bootstrap.cc:492] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: No bootstrap required, opened a new log
I20260924 10:04:33.267750 27331 ts_tablet_manager.cc:1397] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 10:04:33.271970 27330 raft_consensus.cc:348] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.272382 27330 raft_consensus.cc:374] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:33.272514 27330 raft_consensus.cc:729] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9e1bd39ac94b92a25899eb8fa2889c, State: Initialized, Role: FOLLOWER
I20260924 10:04:33.273104 27330 consensus_queue.cc:260] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.274168 27023 heartbeater.cc:503] Master 127.13.162.62:35687 was elected leader, sending a full tablet report...
I20260924 10:04:33.275735 27332 raft_consensus.cc:348] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.276192 27330 ts_tablet_manager.cc:1428] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 10:04:33.276286 27332 raft_consensus.cc:374] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:33.276439 27332 raft_consensus.cc:729] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9613ae4e7324d40a40ef44b304d2a47, State: Initialized, Role: FOLLOWER
I20260924 10:04:33.277101 27332 consensus_queue.cc:260] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.278287 27302 heartbeater.cc:503] Master 127.13.162.62:35687 was elected leader, sending a full tablet report...
I20260924 10:04:33.277997 27331 raft_consensus.cc:348] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.278455 27331 raft_consensus.cc:374] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:33.278544 27331 raft_consensus.cc:729] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88a0d9a238f488d88e9b120e54bdc70, State: Initialized, Role: FOLLOWER
I20260924 10:04:33.279021 27332 ts_tablet_manager.cc:1428] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260924 10:04:33.279167 27331 consensus_queue.cc:260] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.281121 27331 ts_tablet_manager.cc:1428] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260924 10:04:33.383764 27303 tablet.cc:2369] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:04:33.429898 27169 tablet.cc:2369] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:04:33.436877 27337 raft_consensus.cc:482] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:04:33.437181 27337 raft_consensus.cc:504] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.438922 27337 leader_election.cc:288] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb9e1bd39ac94b92a25899eb8fa2889c (127.13.162.1:44857), a88a0d9a238f488d88e9b120e54bdc70 (127.13.162.2:37335)
I20260924 10:04:33.444380 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.439201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:37335 (local address 127.13.162.3:58471)
0924 10:04:33.439496 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:33.439519 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:33.439657 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:33.440062 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:33.440067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:33.440095 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:33.440280 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.440289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.441250 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.441255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.442629 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.442648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.443702 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.443708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.443870 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.443894 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:04:33.443971 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:04:33.444099 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":3}
I20260924 10:04:33.444372 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.439421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:58471 (local address 127.13.162.2:37335)
0924 10:04:33.439619 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.439624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.439649 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:04:33.439856 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.439862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.439939 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.440026 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:33.440410 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.441119 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.442784 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.443556 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.443705 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.443780 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.444012 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.444133 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:04:33.444880 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.439477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:42501 (local address 127.13.162.1:44857)
0924 10:04:33.439830 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.439835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.440128 (+ 293us) server_negotiation.cc:407] Connection header received
0924 10:04:33.440298 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.440302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.440368 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.440441 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:33.440917 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.441622 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.443321 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.444060 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.444208 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.444275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.444565 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.444692 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260924 10:04:33.445439 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.439201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44857 (local address 127.13.162.3:42501)
0924 10:04:33.440012 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:33.440026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:33.440168 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:33.440471 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:33.440475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:33.440497 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:33.440781 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.440789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.441763 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.441770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.443178 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:33.443189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.444249 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.444255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:33.444397 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.444419 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:04:33.445053 (+ 634us) client_negotiation.cc:240] Negotiation successful
0924 10:04:33.445179 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":74,"thread_start_us":461,"threads_started":1}
I20260924 10:04:33.445384 27122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b75c8cbdef40cfa58eb0277a6522f4" candidate_uuid: "d9613ae4e7324d40a40ef44b304d2a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88a0d9a238f488d88e9b120e54bdc70" is_pre_election: true
I20260924 10:04:33.445940 27122 raft_consensus.cc:2454] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9613ae4e7324d40a40ef44b304d2a47 in term 0.
I20260924 10:04:33.446393 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b75c8cbdef40cfa58eb0277a6522f4" candidate_uuid: "d9613ae4e7324d40a40ef44b304d2a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" is_pre_election: true
I20260924 10:04:33.446681 27193 leader_election.cc:302] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88a0d9a238f488d88e9b120e54bdc70, d9613ae4e7324d40a40ef44b304d2a47; no voters:
I20260924 10:04:33.446842 26978 raft_consensus.cc:2454] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9613ae4e7324d40a40ef44b304d2a47 in term 0.
I20260924 10:04:33.447045 27337 raft_consensus.cc:2781] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:04:33.447172 27337 raft_consensus.cc:482] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:04:33.447276 27337 raft_consensus.cc:3037] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:33.449553 27337 raft_consensus.cc:504] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.450131 27337 leader_election.cc:288] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [CANDIDATE]: Term 1 election: Requested vote from peers bb9e1bd39ac94b92a25899eb8fa2889c (127.13.162.1:44857), a88a0d9a238f488d88e9b120e54bdc70 (127.13.162.2:37335)
I20260924 10:04:33.450598 26978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b75c8cbdef40cfa58eb0277a6522f4" candidate_uuid: "d9613ae4e7324d40a40ef44b304d2a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c"
I20260924 10:04:33.450693 27122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b75c8cbdef40cfa58eb0277a6522f4" candidate_uuid: "d9613ae4e7324d40a40ef44b304d2a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88a0d9a238f488d88e9b120e54bdc70"
I20260924 10:04:33.450814 26978 raft_consensus.cc:3037] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:33.450884 27122 raft_consensus.cc:3037] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:33.453902 27122 raft_consensus.cc:2454] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9613ae4e7324d40a40ef44b304d2a47 in term 1.
I20260924 10:04:33.453902 26978 raft_consensus.cc:2454] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9613ae4e7324d40a40ef44b304d2a47 in term 1.
I20260924 10:04:33.454362 27189 leader_election.cc:302] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a88a0d9a238f488d88e9b120e54bdc70, bb9e1bd39ac94b92a25899eb8fa2889c, d9613ae4e7324d40a40ef44b304d2a47; no voters:
I20260924 10:04:33.454685 27337 raft_consensus.cc:2781] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:04:33.455080 27337 raft_consensus.cc:686] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 1 LEADER]: Becoming Leader. State: Replica: d9613ae4e7324d40a40ef44b304d2a47, State: Running, Role: LEADER
I20260924 10:04:33.455526 27337 consensus_queue.cc:237] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } }
I20260924 10:04:33.460496 26824 catalog_manager.cc:5495] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9613ae4e7324d40a40ef44b304d2a47 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "d9613ae4e7324d40a40ef44b304d2a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9613ae4e7324d40a40ef44b304d2a47" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 } health_report { overall_health: UNKNOWN } } }
I20260924 10:04:33.480109 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.475020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.13.162.3:45915)
0924 10:04:33.475229 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.475234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.475253 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:04:33.475349 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.475353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.475437 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.475561 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:04:33.475948 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.476627 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.478062 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.478507 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.478604 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.479084 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:04:33.479606 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:04:33.479742 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.479786 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.479855 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:04:33.488396 27122 raft_consensus.cc:1261] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 1 FOLLOWER]: Refusing update from remote peer d9613ae4e7324d40a40ef44b304d2a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:33.488395 26978 raft_consensus.cc:1261] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 1 FOLLOWER]: Refusing update from remote peer d9613ae4e7324d40a40ef44b304d2a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:33.489315 27340 consensus_queue.cc:1035] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88a0d9a238f488d88e9b120e54bdc70" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:33.489616 27337 consensus_queue.cc:1035] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9e1bd39ac94b92a25899eb8fa2889c" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:33.517132 26824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:04:33.517445 26824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:04:33.521678 26824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e387e67b69e444f0b8461d9ddca87b4c: Sending DeleteTablet for 3 replicas of tablet 03b75c8cbdef40cfa58eb0277a6522f4
I20260924 10:04:33.522881 26958 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b75c8cbdef40cfa58eb0277a6522f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:33 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:58536
I20260924 10:04:33.523309 27237 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b75c8cbdef40cfa58eb0277a6522f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:33 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:55670
I20260924 10:04:33.523326 27102 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b75c8cbdef40cfa58eb0277a6522f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:33 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:47278
I20260924 10:04:33.523761 27353 tablet_replica.cc:330] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: stopping tablet replica
I20260924 10:04:33.524125 27353 raft_consensus.cc:2229] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:04:33.524637 27353 raft_consensus.cc:2258] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:33.531596 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 26863
I20260924 10:04:33.535754 27353 ts_tablet_manager.cc:1905] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:33.541891 27353 ts_tablet_manager.cc:1918] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:33.542198 27353 log.cc:1196] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/03b75c8cbdef40cfa58eb0277a6522f4
I20260924 10:04:33.542912 27353 ts_tablet_manager.cc:1939] T 03b75c8cbdef40cfa58eb0277a6522f4 P bb9e1bd39ac94b92a25899eb8fa2889c: Deleting consensus metadata
I20260924 10:04:33.545083 26810 catalog_manager.cc:4841] TS bb9e1bd39ac94b92a25899eb8fa2889c (127.13.162.1:44857): tablet 03b75c8cbdef40cfa58eb0277a6522f4 (table test-table [id=fcd40d8e0ec5481aa294ba9aa638755f]) successfully deleted
I20260924 10:04:33.547953 27355 tablet_replica.cc:330] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: stopping tablet replica
I20260924 10:04:33.549639 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.532691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.13.162.1:44857)
0924 10:04:33.532878 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.532884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.532978 (+ 94us) server_negotiation.cc:407] Connection header received
0924 10:04:33.533165 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.533170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.533263 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.533376 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:33.535217 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.536257 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.538372 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.538997 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.539120 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.542539 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:33.542554 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:33.542573 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:33.542614 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:33.546644 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.547340 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.547349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.547352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.547525 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.547914 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.547920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.547923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.548377 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:33.548676 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.548991 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.549284 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:04:33.549695 27355 raft_consensus.cc:2229] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:04:33.550434 27355 raft_consensus.cc:2258] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:33.550716 27018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:33.553241 27355 ts_tablet_manager.cc:1905] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:33.553993 27354 tablet_replica.cc:330] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: stopping tablet replica
I20260924 10:04:33.554906 27354 raft_consensus.cc:2229] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:04:33.555784 27354 raft_consensus.cc:2258] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:33.559664 27354 ts_tablet_manager.cc:1905] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:33.564100 27355 ts_tablet_manager.cc:1918] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:33.564399 27355 log.cc:1196] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/03b75c8cbdef40cfa58eb0277a6522f4
I20260924 10:04:33.565008 27355 ts_tablet_manager.cc:1939] T 03b75c8cbdef40cfa58eb0277a6522f4 P d9613ae4e7324d40a40ef44b304d2a47: Deleting consensus metadata
I20260924 10:04:33.566870 27354 ts_tablet_manager.cc:1918] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:33.567026 26813 catalog_manager.cc:4841] TS d9613ae4e7324d40a40ef44b304d2a47 (127.13.162.3:45915): tablet 03b75c8cbdef40cfa58eb0277a6522f4 (table test-table [id=fcd40d8e0ec5481aa294ba9aa638755f]) successfully deleted
I20260924 10:04:33.567107 27354 log.cc:1196] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/03b75c8cbdef40cfa58eb0277a6522f4
I20260924 10:04:33.567690 27354 ts_tablet_manager.cc:1939] T 03b75c8cbdef40cfa58eb0277a6522f4 P a88a0d9a238f488d88e9b120e54bdc70: Deleting consensus metadata
I20260924 10:04:33.569509 26813 catalog_manager.cc:4841] TS a88a0d9a238f488d88e9b120e54bdc70 (127.13.162.2:37335): tablet 03b75c8cbdef40cfa58eb0277a6522f4 (table test-table [id=fcd40d8e0ec5481aa294ba9aa638755f]) successfully deleted
I20260924 10:04:33.646291 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 26863
I20260924 10:04:33.660090 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27027
I20260924 10:04:33.673585 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.661007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.13.162.2:37335)
0924 10:04:33.661207 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.661212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.661235 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:04:33.661434 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.661439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.661500 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.661580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:33.662910 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.663708 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.664958 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.665350 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.665458 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.667998 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:33.668012 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:33.668027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:33.668064 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:33.670882 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.671583 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.671591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.671593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.671752 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.672096 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.672100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.672102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.672446 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:33.672732 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.673008 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.673221 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:04:33.674227 27163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:33.761476 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27027
I20260924 10:04:33.780300 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27172
I20260924 10:04:33.796186 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:33.781110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.13.162.3:45915)
0924 10:04:33.781440 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:33.781446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:33.781518 (+ 72us) server_negotiation.cc:407] Connection header received
0924 10:04:33.781616 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:33.781622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:33.781742 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:33.781852 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:33.783232 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.784412 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.785694 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:33.786402 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:33.786543 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:33.788973 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:33.788987 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:33.789008 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:33.789049 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:33.793034 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.793770 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.793779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.793783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.793951 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:33.794279 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:33.794284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:33.794287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:33.794919 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:33.795238 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:33.795439 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 10:04:33.795793 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 10:04:33.796880 27297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:33.917541 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27172
I20260924 10:04:33.932657 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 26793
I20260924 10:04:33.933749 26856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:34.043038 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 26793
I20260924 10:04:34.062353 13960 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:04:34.397894 13960 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:04:34.398095 13960 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:04:34.733451 13960 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:04:34.733 UTC [25845] LOG: received smart shutdown request
2026-09-24 10:04:34.736 UTC [25845] LOG: background worker "logical replication launcher" (PID 25854) exited with exit code 1
2026-09-24 10:04:34.736 UTC [25849] LOG: shutting down
2026-09-24 10:04:34.882 UTC [25845] LOG: database system is shut down
2026-09-24T10:04:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50452 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:04:35 dist-test-slave-jvsj krb5kdc[27368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:04:35 dist-test-slave-jvsj krb5kdc[27368](info): set up 2 sockets
Sep 24 10:04:35 dist-test-slave-jvsj krb5kdc[27368](info): commencing operation
krb5kdc: starting...
W20260924 10:04:35.986534 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:04:36 dist-test-slave-jvsj krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:04:36.201829 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36261
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36261,127.13.162.61:46261,127.13.162.60:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:36.525009 27381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:36.525301 27381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:36.525434 27381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:36.532706 27381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:36.532823 27381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:36.532869 27381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:36.532923 27381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:36.532965 27381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:36.542847 27381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36261,127.13.162.61:46261,127.13.162.60:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:36.544128 27381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:36.545356 27381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:36.556927 27386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:36.556957 27387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:36.556993 27389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:36.557705 27381 server_base.cc:1034] running on GCE node
I20260924 10:04:36.558591 27381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:36.560030 27381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:36.561246 27381 hybrid_clock.cc:648] HybridClock initialized: now 1790244276561181 us; error 42 us; skew 500 ppm
I20260924 10:04:36.561568 27381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:36 dist-test-slave-jvsj krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:36.564543 27381 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:36.565953 27381 webserver.cc:460] Webserver started at http://127.13.162.62:40657/ using document root <none> and password file <none>
I20260924 10:04:36.566677 27381 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:36.566792 27381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:36.567073 27381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:36.569994 27381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "3aca5cb285014bdbb6c57ed5915d0989"
format_stamp: "Formatted at 2026-09-24 10:04:36 on dist-test-slave-jvsj"
server_key: "59d3db313c0bd4a415bebc8ae66918c6"
server_key_iv: "2d9999cf792d477f0258e25607149be2"
server_key_version: "encryptionkey@0"
I20260924 10:04:36.570804 27381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "3aca5cb285014bdbb6c57ed5915d0989"
format_stamp: "Formatted at 2026-09-24 10:04:36 on dist-test-slave-jvsj"
server_key: "59d3db313c0bd4a415bebc8ae66918c6"
server_key_iv: "2d9999cf792d477f0258e25607149be2"
server_key_version: "encryptionkey@0"
I20260924 10:04:36.575860 27381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 10:04:36.579350 27396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:36.580762 27381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:04:36.580943 27381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "3aca5cb285014bdbb6c57ed5915d0989"
format_stamp: "Formatted at 2026-09-24 10:04:36 on dist-test-slave-jvsj"
server_key: "59d3db313c0bd4a415bebc8ae66918c6"
server_key_iv: "2d9999cf792d477f0258e25607149be2"
server_key_version: "encryptionkey@0"
I20260924 10:04:36.581142 27381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:36.597993 27381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:36.602543 27381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:36.602821 27381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:36.620370 27381 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:36261
I20260924 10:04:36.620393 27447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:36261 every 8 connection(s)
I20260924 10:04:36.621773 27381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:36.625610 27448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:36.632180 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27381
I20260924 10:04:36.632421 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:04:36.633051 13960 external_mini_cluster.cc:1428] Setting key 73f9f11b1621fe8e3f9496a0cc4332ec
I20260924 10:04:36.634862 27448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:04:36 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.61@KRBTEST.COM" created.
I20260924 10:04:36.659081 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.638201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36261 (local address 127.0.0.1:53026)
0924 10:04:36.639191 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.639254 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:36.641324 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:36.642338 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:36.642345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:36.642911 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:36.643254 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:36.643272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:36.645373 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:36.645377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:36.646537 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:36.646549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:36.647901 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:36.647911 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:36.648180 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:36.648890 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:36.648913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:36.651350 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:36.654926 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:36.654936 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:36.654947 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:36.655404 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:36.655923 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:36.655926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:36.655928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:36.656082 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:36.657041 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:36.657051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:36.657280 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 10:04:36.657493 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 10:04:36.657906 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":395,"threads_started":1}
I20260924 10:04:36.659111 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.638201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.13.162.62:36261)
0924 10:04:36.640192 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:36.640198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:36.640246 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:04:36.641741 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:36.641749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:36.642060 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:36.642268 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:36.643443 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:36.645219 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:36.646674 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:36.647543 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:36.647717 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:36.651623 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:36.651635 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:36.651653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:36.651689 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:36.654791 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:36.655555 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:36.655564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:36.655568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:36.655778 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:36.656202 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:36.656206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:36.656210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:36.656899 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:36.657095 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:36.657683 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 10:04:36.658071 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":1797,"spinlock_wait_cycles":37248,"thread_start_us":1479,"threads_started":1}
I20260924 10:04:36.662147 27448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:04:36.663442 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.663018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33632)
0924 10:04:36.663224 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.663332 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 10:04:36.663746 27400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
I20260924 10:04:36.665239 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.664846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33636)
0924 10:04:36.665015 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.665094 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:04:36.665436 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:46261: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab.
Entry for principal kudu/127.13.162.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab.
I20260924 10:04:36.683377 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:46261
--webserver_interface=127.13.162.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab
--principal=kudu/127.13.162.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36261,127.13.162.61:46261,127.13.162.60:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:04:36.714793 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } attempt: 1
I20260924 10:04:36.716059 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.715520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33640)
0924 10:04:36.715751 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.715889 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 10:04:36.717146 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.716781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33644)
0924 10:04:36.716933 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.717010 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 10:04:36.717348 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:46261: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
I20260924 10:04:36.785601 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } attempt: 2
I20260924 10:04:36.786703 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.786210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33660)
0924 10:04:36.786469 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.786562 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 10:04:36.787635 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.787292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33672)
0924 10:04:36.787433 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.787513 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:04:36.787815 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:46261: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
I20260924 10:04:36.879113 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } attempt: 3
I20260924 10:04:36.880252 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.879813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33676)
0924 10:04:36.880019 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.880108 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:04:36.881242 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:36.880867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33686)
0924 10:04:36.881017 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:36.881101 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:04:36.881433 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:46261: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
W20260924 10:04:37.001367 27455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:37.001693 27455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:37.001852 27455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:37.009166 27455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:37.009282 27455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:37.009328 27455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:37.009383 27455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:37.009425 27455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:37.019534 27455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36261,127.13.162.61:46261,127.13.162.60:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab
--principal=kudu/127.13.162.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.162.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:37.021521 27455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:37.023265 27455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:04:37.024645 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } attempt: 4
I20260924 10:04:37.025801 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.025246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33700)
0924 10:04:37.025521 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.025642 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260924 10:04:37.026947 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.026531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33706)
0924 10:04:37.026685 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.026792 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:04:37.027235 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:46261: Network error: Client connection negotiation failed: client connection to 127.13.162.61:46261: connect: Connection refused (error 111)
W20260924 10:04:37.035048 27460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:37.035916 27461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:37.036727 27463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:37.037170 27455 server_base.cc:1034] running on GCE node
I20260924 10:04:37.037997 27455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:37.039528 27455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:37.040757 27455 hybrid_clock.cc:648] HybridClock initialized: now 1790244277040693 us; error 58 us; skew 500 ppm
I20260924 10:04:37.041105 27455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:37.044286 27455 init.cc:377] Logged in from keytab as kudu/127.13.162.61@KRBTEST.COM (short username kudu)
I20260924 10:04:37.045759 27455 webserver.cc:460] Webserver started at http://127.13.162.61:34189/ using document root <none> and password file <none>
I20260924 10:04:37.046506 27455 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:37.046628 27455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:37.046942 27455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:37.049877 27455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data/instance:
uuid: "069190afd3254a15ba53c59165d1d04b"
format_stamp: "Formatted at 2026-09-24 10:04:37 on dist-test-slave-jvsj"
server_key: "54f96dc1cd4f1183cdc8b80d64cbbf7b"
server_key_iv: "5286dff09211b77c3943ee5f8132759a"
server_key_version: "encryptionkey@0"
I20260924 10:04:37.050711 27455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal/instance:
uuid: "069190afd3254a15ba53c59165d1d04b"
format_stamp: "Formatted at 2026-09-24 10:04:37 on dist-test-slave-jvsj"
server_key: "54f96dc1cd4f1183cdc8b80d64cbbf7b"
server_key_iv: "5286dff09211b77c3943ee5f8132759a"
server_key_version: "encryptionkey@0"
I20260924 10:04:37.055786 27455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 10:04:37.059110 27470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:37.060428 27455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:04:37.060582 27455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal
uuid: "069190afd3254a15ba53c59165d1d04b"
format_stamp: "Formatted at 2026-09-24 10:04:37 on dist-test-slave-jvsj"
server_key: "54f96dc1cd4f1183cdc8b80d64cbbf7b"
server_key_iv: "5286dff09211b77c3943ee5f8132759a"
server_key_version: "encryptionkey@0"
I20260924 10:04:37.060765 27455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:37.078439 27455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:37.082865 27455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:37.083242 27455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:37.100589 27455 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.61:46261
I20260924 10:04:37.100690 27521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.61:46261 every 8 connection(s)
I20260924 10:04:37.101934 27455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
I20260924 10:04:37.103744 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27455
I20260924 10:04:37.103899 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-1/wal/instance
I20260924 10:04:37.104353 13960 external_mini_cluster.cc:1428] Setting key 7ed347ebe7653ba9e7e292274ee19551
I20260924 10:04:37.105832 27522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:37.113209 27522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.162.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.60@KRBTEST.COM" created.
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:37.134508 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.116498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.13.162.62:36261)
0924 10:04:37.116713 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.116720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.118031 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 10:04:37.120107 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.120113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.120194 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.120283 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.121895 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.122620 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.124176 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.124585 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.124698 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.128609 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.128645 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.128648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.128727 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.131318 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.132240 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.132244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.132247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.132403 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.132765 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.132770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.132773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.133106 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.133246 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.133948 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.134179 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:04:37.135380 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.117023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36261 (local address 127.0.0.1:53034)
0924 10:04:37.117852 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.117902 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.119817 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.120532 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.120541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.121113 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.121712 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.121730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.122785 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.122790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.124019 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.124040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.124809 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.124816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.125046 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.125743 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.125773 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.128360 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.131464 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.131475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.131493 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.132108 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.132505 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.132509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.132511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.132675 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.133205 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.133211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.133532 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.133825 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.134211 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260924 10:04:37.137193 27522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab.
Entry for principal kudu/127.13.162.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab.
I20260924 10:04:37.153038 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:43035
--webserver_interface=127.13.162.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab
--principal=kudu/127.13.162.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:36261,127.13.162.61:46261,127.13.162.60:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:04:37.153110 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.137957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33714)
0924 10:04:37.138174 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.138195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.138291 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.139181 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.139189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.139649 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.139841 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.139851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.141264 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.141270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.142618 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.142633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.143478 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.143485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.143702 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.144346 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.144369 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.147413 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.150585 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.150591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.150595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.150966 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.151364 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.151369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.151372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.151501 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.152188 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.152192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.152336 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.152453 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.152734 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 10:04:37.153220 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.138119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.13.162.61:46261)
0924 10:04:37.138777 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.138783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.138808 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:04:37.138848 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.138854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.139029 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.139137 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.139983 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.141124 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.142782 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.143322 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.143423 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.147648 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.147667 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.147685 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.147724 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.150471 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.151111 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.151118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.151120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.151253 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.151605 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.151610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.151613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.152053 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.152259 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.152639 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.152864 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":381,"threads_started":1}
I20260924 10:04:37.155834 27522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:04:37.157004 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.156481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45918)
0924 10:04:37.156662 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.156839 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 10:04:37.157275 27471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.158874 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.158532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45932)
0924 10:04:37.158673 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.158764 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:04:37.159085 27522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.208438 27522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 1
I20260924 10:04:37.209601 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.209135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45948)
0924 10:04:37.209332 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.209443 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 10:04:37.210631 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.210221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45958)
0924 10:04:37.210422 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.210517 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 10:04:37.210847 27522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.279083 27522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 2
I20260924 10:04:37.280346 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.279859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45974)
0924 10:04:37.280073 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.280180 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 10:04:37.281374 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.281034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45988)
0924 10:04:37.281183 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.281254 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 10:04:37.281582 27522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.326435 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } attempt: 5
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:04:37.339845 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.327335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.13.162.61:46261)
0924 10:04:37.327527 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.327532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.327552 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:04:37.327608 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.327611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.327695 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.327775 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.328580 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.329269 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.330446 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.331158 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.331321 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.334559 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.334565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.334570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.334613 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.337334 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.337946 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.337952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.337956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.338079 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.338485 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.338490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.338494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.338786 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.338917 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.339299 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.339503 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:04:37.339845 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.327116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33716)
0924 10:04:37.327307 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.327329 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.327454 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.327835 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.327841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.328249 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.328422 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.328433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.329395 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.329400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.330272 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.330284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.331327 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.331331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.331466 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.332087 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.332106 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.334302 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.337462 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.337467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.337481 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.337839 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.338187 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.338192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.338196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.338380 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.338896 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.338901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.339050 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.339174 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.339485 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:04:37.341194 27448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:04:37.342285 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.341901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:45998)
0924 10:04:37.342062 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.342138 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:04:37.343225 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.342891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46000)
0924 10:04:37.343037 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.343105 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:04:37.343405 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.372792 27522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 3
I20260924 10:04:37.374020 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.373477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46016)
0924 10:04:37.373755 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.373858 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 10:04:37.375033 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.374641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46030)
0924 10:04:37.374847 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.374921 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 10:04:37.375216 27522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.394618 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 1
I20260924 10:04:37.395706 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.395245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46044)
0924 10:04:37.395433 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.395525 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:04:37.396740 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.396397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46060)
0924 10:04:37.396549 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.396624 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:04:37.396939 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.465165 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 2
I20260924 10:04:37.466347 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.465824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46074)
0924 10:04:37.466040 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.466150 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:04:37.467664 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.467179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46084)
0924 10:04:37.467409 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.467508 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 10:04:37.467876 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
W20260924 10:04:37.474677 27529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:37.474972 27529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:37.475102 27529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:37.482345 27529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:37.482463 27529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:37.482535 27529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:37.482606 27529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:37.482661 27529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:37.492666 27529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:36261,127.13.162.61:46261,127.13.162.60:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab
--principal=kudu/127.13.162.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.162.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:37.494028 27529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:37.495339 27529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:37.506685 27534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:37.506661 27535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:37.506737 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:37.507833 27529 server_base.cc:1034] running on GCE node
I20260924 10:04:37.508661 27529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:37.510135 27529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:37.511389 27529 hybrid_clock.cc:648] HybridClock initialized: now 1790244277511327 us; error 47 us; skew 500 ppm
I20260924 10:04:37.511794 27529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:37.515086 27529 init.cc:377] Logged in from keytab as kudu/127.13.162.60@KRBTEST.COM (short username kudu)
I20260924 10:04:37.516812 27529 webserver.cc:460] Webserver started at http://127.13.162.60:40095/ using document root <none> and password file <none>
I20260924 10:04:37.517537 27529 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:37.517663 27529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:37.517954 27529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:37.518396 27522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 4
I20260924 10:04:37.519646 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.519119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46088)
0924 10:04:37.519368 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.519465 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:04:37.520635 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.520321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46100)
0924 10:04:37.520456 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.520526 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:04:37.520876 27522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.521827 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data/instance:
uuid: "936cb4aeec62448fb01f337502fcceed"
format_stamp: "Formatted at 2026-09-24 10:04:37 on dist-test-slave-jvsj"
server_key: "7f2f4afae8157ff5b9d782981774cc51"
server_key_iv: "425842dff85db6133c1bd17b70c95fb9"
server_key_version: "encryptionkey@0"
I20260924 10:04:37.522680 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal/instance:
uuid: "936cb4aeec62448fb01f337502fcceed"
format_stamp: "Formatted at 2026-09-24 10:04:37 on dist-test-slave-jvsj"
server_key: "7f2f4afae8157ff5b9d782981774cc51"
server_key_iv: "425842dff85db6133c1bd17b70c95fb9"
server_key_version: "encryptionkey@0"
I20260924 10:04:37.528113 27529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:04:37.531857 27544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:37.533397 27529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:04:37.533586 27529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal
uuid: "936cb4aeec62448fb01f337502fcceed"
format_stamp: "Formatted at 2026-09-24 10:04:37 on dist-test-slave-jvsj"
server_key: "7f2f4afae8157ff5b9d782981774cc51"
server_key_iv: "425842dff85db6133c1bd17b70c95fb9"
server_key_version: "encryptionkey@0"
I20260924 10:04:37.533869 27529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:37.561713 27529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:37.565979 27529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:37.566228 27529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:37.574102 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 3
I20260924 10:04:37.575341 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.574835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46104)
0924 10:04:37.575066 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.575172 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 10:04:37.576574 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.576126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46112)
0924 10:04:37.576354 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.576427 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260924 10:04:37.576824 27448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:43035: Network error: Client connection negotiation failed: client connection to 127.13.162.60:43035: connect: Connection refused (error 111)
I20260924 10:04:37.583130 27529 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.60:43035
I20260924 10:04:37.583127 27595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.60:43035 every 8 connection(s)
I20260924 10:04:37.584842 27529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
I20260924 10:04:37.585255 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27529
I20260924 10:04:37.585454 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244085964582-13960-0/minicluster-data/master-2/wal/instance
I20260924 10:04:37.586023 13960 external_mini_cluster.cc:1428] Setting key 550560d0c23f55df93fda8b23d5ee67b
I20260924 10:04:37.589032 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:37.597167 27596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:37.609647 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.587943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.13.162.60:43035)
0924 10:04:37.588710 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.588723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.588764 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:04:37.590508 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.590531 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.590918 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.591236 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.593006 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.594495 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.596362 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.597110 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.597267 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.601930 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.601939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.601949 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.601988 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.605140 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.605979 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.605985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.605992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.606164 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.606605 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.606610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.606612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.607209 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.607443 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.607968 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.608426 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":341,"threads_started":1}
I20260924 10:04:37.614589 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.600168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:36261 (local address 127.0.0.1:53044)
0924 10:04:37.600927 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.600969 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.601123 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.601467 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.601474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.602083 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.602447 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.602459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.603803 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.603809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.604961 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.604973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.605837 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.605843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.606018 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.606628 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.606646 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.608945 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.611983 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.611993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.612000 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.612403 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.612897 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.612900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.612902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.613013 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.613520 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.613527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.613707 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.613851 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.614144 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":390,"threads_started":1}
I20260924 10:04:37.614609 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.600213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.13.162.62:36261)
0924 10:04:37.600386 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.600390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.601059 (+ 669us) server_negotiation.cc:407] Connection header received
0924 10:04:37.601242 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.601247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.601313 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.601393 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.602592 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.603624 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.605119 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.605652 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.605750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.609138 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.609145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.609149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.609190 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.611869 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.612537 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.612542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.612546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.612731 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.613117 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.613122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.613125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.613401 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.613539 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.613975 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.614166 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260924 10:04:37.615962 27596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:04:37.628844 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.612454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.13.162.62:36261)
0924 10:04:37.613009 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.613014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.613031 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:04:37.613066 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.613069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.613118 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.613214 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.614404 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.615487 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.616958 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.617506 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.617628 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.622580 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.622587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.622592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.622632 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.625591 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.626294 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.626300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.626304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.626467 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.626817 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.626822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.626826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.627111 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.627344 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.628132 (+ 788us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.628340 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":454,"thread_start_us":260,"threads_started":1}
I20260924 10:04:37.629827 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.616725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:46261 (local address 127.0.0.1:33720)
0924 10:04:37.616879 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.616895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.616982 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.617288 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.617293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.617698 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.617849 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.617857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.619160 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.619164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.620091 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.620103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.620760 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.620765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.620936 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.621403 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.621426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.623609 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.627239 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.627244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.627248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.627632 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.628045 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.628050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.628054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.628221 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.628865 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.628869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.629033 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.629210 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.629495 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:04:37.630110 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.616745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.13.162.61:46261)
0924 10:04:37.616922 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.616928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.616946 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:04:37.617083 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.617087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.617149 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.617242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.617990 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.619030 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.620218 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.620623 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.620730 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.623813 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.623821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.623826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.623866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.627098 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.627748 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.627754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.627758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.627913 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.628335 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.628340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.628344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.628648 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.628775 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.629331 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.629553 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:04:37.631151 27596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:04:37.644250 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.631930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46130)
0924 10:04:37.632226 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.632243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.632359 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.632620 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.632628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.632956 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.633086 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.633093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.634142 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.634147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.635349 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.635360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.636077 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.636082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.636260 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.636727 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.636749 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.639096 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.641822 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.641827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.641831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.642284 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.642616 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.642620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.642624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.642776 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.643343 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.643347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.643494 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.643622 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.643869 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260924 10:04:37.644325 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.631941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.13.162.60:43035)
0924 10:04:37.632114 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.632119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.632310 (+ 191us) server_negotiation.cc:407] Connection header received
0924 10:04:37.632448 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.632452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.632505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.632572 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.633210 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.634020 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.635465 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.635927 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.636038 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.639249 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.639254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.639258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.639285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.641703 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.642402 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.642405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.642407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.642521 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.642888 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.642892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.642895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.643239 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.643381 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.643734 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.644034 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":46}
I20260924 10:04:37.645588 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.630781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.13.162.61:46261)
0924 10:04:37.630908 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.630912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.630925 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:04:37.630959 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.630963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.631029 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.631121 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.632672 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.633508 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.635041 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.635434 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.635550 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.640006 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.640011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.640015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.640046 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.642657 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.643385 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.643390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.643393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.643544 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.643957 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.643962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.643966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.644274 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.644525 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.645059 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.645330 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:04:37.649232 27596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: Bootstrap starting.
I20260924 10:04:37.649852 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:04:37.652312 27596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:37.653458 27596 log.cc:824] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:37.660280 27596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: No bootstrap required, opened a new log
I20260924 10:04:37.670893 27596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.671563 27596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:37.671742 27596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936cb4aeec62448fb01f337502fcceed, State: Initialized, Role: FOLLOWER
I20260924 10:04:37.672667 27596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.675910 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.654806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.13.162.62:36261)
0924 10:04:37.655013 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.655018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.657598 (+ 2580us) server_negotiation.cc:407] Connection header received
0924 10:04:37.657855 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.657861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.657946 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.658017 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.660622 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.661730 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.663990 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.664568 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.664663 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.668679 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.668686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.668691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.668734 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.672461 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.673476 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.673481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.673485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.673660 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.674105 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.674110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.674114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.674505 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.674747 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.675319 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.675559 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:04:37.676090 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.654406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.13.162.60:43035)
0924 10:04:37.654617 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.654623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.656663 (+ 2040us) server_negotiation.cc:407] Connection header received
0924 10:04:37.657677 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.657693 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.657771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.657842 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.660114 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.661154 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.663837 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.664620 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.664729 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.668855 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.668863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.668868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.668904 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.672299 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.673391 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.673398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.673402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.673593 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.674188 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.674194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.674198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.674650 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.674915 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.675471 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.675739 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:04:37.677242 27610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.677661 27610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.678702 27596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:37.680405 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.656822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.13.162.61:46261)
0924 10:04:37.656995 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.657001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.663653 (+ 6652us) server_negotiation.cc:407] Connection header received
0924 10:04:37.663823 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.663828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.663911 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.664014 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.666815 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.667860 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.669709 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.670309 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.670424 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.673724 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.673730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.673735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.673777 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.677058 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.678414 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.678419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.678423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.678588 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.679044 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.679048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.679050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.679339 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.679571 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.679873 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.680091 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:04:37.689400 27621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:04:37.689525 27621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:04:37.685849 27529 master_runner.cc:186] Error getting master registration for 127.13.162.62:36261: OK, instance_id { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" instance_seqno: 1790244276615147 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:04:37.693864 27529 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:04:37.702175 27610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:04:37.702493 27610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.704180 27610 leader_election.cc:288] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aca5cb285014bdbb6c57ed5915d0989 (127.13.162.62:36261), 069190afd3254a15ba53c59165d1d04b (127.13.162.61:46261)
I20260924 10:04:37.754038 27448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 4
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244276, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
I20260924 10:04:37.770594 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.754607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46142)
0924 10:04:37.754818 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.754847 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.754974 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.755457 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.755463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.755964 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.756165 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.756177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.757644 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.757650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.758707 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.758720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.759674 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.759680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.759858 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.760466 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.760490 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.764653 (+ 4163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.768126 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.768132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.768136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.768661 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.769002 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.769007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.769011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.769182 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.769829 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.769832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.769957 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.770066 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.770273 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:04:37.771451 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.754657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.13.162.60:43035)
0924 10:04:37.754847 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.754853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.754903 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:04:37.755088 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.755093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.755173 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.755266 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.756328 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.757512 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.758846 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.759517 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.759634 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.764852 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.764858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.764863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.764902 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.768008 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.768768 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.768771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.768774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.768912 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.769321 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.769327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.769331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.769712 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.769867 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.770200 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.770450 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:04:37.776978 27448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: Bootstrap starting.
I20260924 10:04:37.780716 27448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:37.781747 27448 log.cc:824] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:37.784437 27448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: No bootstrap required, opened a new log
I20260924 10:04:37.791740 27448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.792104 27448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:37.792223 27448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aca5cb285014bdbb6c57ed5915d0989, State: Initialized, Role: FOLLOWER
I20260924 10:04:37.792800 27448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.793697 27624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.794154 27624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.795253 27448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:37.796018 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "936cb4aeec62448fb01f337502fcceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aca5cb285014bdbb6c57ed5915d0989" is_pre_election: true
I20260924 10:04:37.796609 27423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 936cb4aeec62448fb01f337502fcceed in term 0.
I20260924 10:04:37.797353 27546 leader_election.cc:302] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aca5cb285014bdbb6c57ed5915d0989, 936cb4aeec62448fb01f337502fcceed; no voters:
I20260924 10:04:37.797806 27610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:04:37.797951 27610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:04:37.798043 27610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:37.801442 27610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.802652 27423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "936cb4aeec62448fb01f337502fcceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aca5cb285014bdbb6c57ed5915d0989"
I20260924 10:04:37.802870 27423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:37.803174 27610 leader_election.cc:288] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [CANDIDATE]: Term 1 election: Requested vote from peers 3aca5cb285014bdbb6c57ed5915d0989 (127.13.162.62:36261), 069190afd3254a15ba53c59165d1d04b (127.13.162.61:46261)
I20260924 10:04:37.805897 27423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 936cb4aeec62448fb01f337502fcceed in term 1.
I20260924 10:04:37.806356 27546 leader_election.cc:302] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aca5cb285014bdbb6c57ed5915d0989, 936cb4aeec62448fb01f337502fcceed; no voters:
I20260924 10:04:37.806648 27610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 1 FOLLOWER]: Leader election won for term 1
W20260924 10:04:37.818404 27550 tablet.cc:2369] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:04:37.820114 27522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } attempt: 5
I20260924 10:04:37.824239 27610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [term 1 LEADER]: Becoming Leader. State: Replica: 936cb4aeec62448fb01f337502fcceed, State: Running, Role: LEADER
I20260924 10:04:37.825198 27610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
Sep 24 10:04:37 dist-test-slave-jvsj krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
W20260924 10:04:37.832402 27636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:04:37.832576 27636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:04:37.811985 27381 master_runner.cc:186] Error getting master registration for 127.13.162.61:46261: OK, instance_id { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" instance_seqno: 1790244277095622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:04:37.838867 27381 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:04:37.839304 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.820733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:43035 (local address 127.0.0.1:46144)
0924 10:04:37.821101 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:37.821123 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:37.821240 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:37.821791 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:37.821797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:37.822392 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:37.822626 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.822639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.824257 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.824262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.825730 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:37.825747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.827167 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.827175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:37.827400 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.828264 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:37.828311 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:37.832169 (+ 3858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:37.836212 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.836219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.836223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.836758 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.837188 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:37.837193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:37.837197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:37.837372 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:37.838123 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:37.838128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:37.838294 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 10:04:37.838464 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 10:04:37.838849 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260924 10:04:37.840224 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:37.821099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.13.162.60:43035)
0924 10:04:37.821287 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:37.821293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:37.821315 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:04:37.821379 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:37.821384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:37.821463 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:37.821582 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:37.822826 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.824051 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.826026 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:37.826811 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:37.826930 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:37.832440 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:37.832447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:37.832451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:37.832489 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:37.836040 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.836897 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.836903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.836906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.837059 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:37.837500 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:37.837505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:37.837508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:37.837920 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:37.838064 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:37.839586 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0924 10:04:37.839839 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:04:37.840960 27635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936cb4aeec62448fb01f337502fcceed. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.841511 27635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:37.847669 27522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: Bootstrap starting.
I20260924 10:04:37.851869 27522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:37.853641 27522 log.cc:824] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: Log is configured to *not* fsync() on all Append() calls
W20260924 10:04:37.855600 27402 tablet.cc:2369] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:04:37.862305 27639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:37.866990 27522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: No bootstrap required, opened a new log
I20260924 10:04:37.871519 27639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:37.881814 27522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.885394 27423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [term 1 FOLLOWER]: Refusing update from remote peer 936cb4aeec62448fb01f337502fcceed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:37.886546 27635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:37.882257 27522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:37.890465 27522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069190afd3254a15ba53c59165d1d04b, State: Initialized, Role: FOLLOWER
I20260924 10:04:37.891301 27522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } }
I20260924 10:04:37.894848 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.895588 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.896317 27522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:37.896306 27497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "936cb4aeec62448fb01f337502fcceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069190afd3254a15ba53c59165d1d04b" is_pre_election: true
I20260924 10:04:37.896663 27495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:37.897174 27496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "936cb4aeec62448fb01f337502fcceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069190afd3254a15ba53c59165d1d04b"
I20260924 10:04:37.900431 27495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [term 1 FOLLOWER]: Refusing update from remote peer 936cb4aeec62448fb01f337502fcceed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:37.902503 27635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:37.923493 27624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936cb4aeec62448fb01f337502fcceed. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.923959 27624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.938787 27610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.939194 27610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:37.941395 27639 catalog_manager.cc:1347] Generated new cluster ID: 2fc54f71fd1c41c7b3ce07cb93175a3b
I20260924 10:04:37.941499 27639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:04:37.945289 27624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.945544 27624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.964936 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936cb4aeec62448fb01f337502fcceed. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.965291 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.967975 27635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.968269 27635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:37.984951 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "936cb4aeec62448fb01f337502fcceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aca5cb285014bdbb6c57ed5915d0989" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 36261 } } peers { permanent_uuid: "069190afd3254a15ba53c59165d1d04b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 46261 } } peers { permanent_uuid: "936cb4aeec62448fb01f337502fcceed" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 43035 } } }
I20260924 10:04:37.985236 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:04:37.996611 27639 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:04:37.998546 27639 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:04:38.016570 27639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 936cb4aeec62448fb01f337502fcceed: Generated new TSK 0
I20260924 10:04:38.017838 27639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:04:38.041015 27660 catalog_manager.cc:1259] Loaded cluster ID: 2fc54f71fd1c41c7b3ce07cb93175a3b
I20260924 10:04:38.041246 27660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: loading cluster ID for follower catalog manager: success
I20260924 10:04:38.052004 27660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: acquiring CA information for follower catalog manager: success
I20260924 10:04:38.056216 27660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 069190afd3254a15ba53c59165d1d04b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:04:38.834643 27636 catalog_manager.cc:1259] Loaded cluster ID: 2fc54f71fd1c41c7b3ce07cb93175a3b
I20260924 10:04:38.834744 27636 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: loading cluster ID for follower catalog manager: success
I20260924 10:04:38.837215 27636 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: acquiring CA information for follower catalog manager: success
I20260924 10:04:38.839233 27636 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3aca5cb285014bdbb6c57ed5915d0989: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:04:39.171079 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27381
I20260924 10:04:39.173388 27443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:39.280388 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27381
I20260924 10:04:39.297739 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27455
I20260924 10:04:39.298949 27517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:39.405009 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27455
W20260924 10:04:39.421268 27548 connection.cc:535] client connection to 127.13.162.61:46261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:04:39.421530 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27529
W20260924 10:04:39.421561 27548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:04:39.422643 27591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:39.530535 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27529
2026-09-24T10:04:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4546 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:04:39 dist-test-slave-jvsj krb5kdc[27674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:04:39 dist-test-slave-jvsj krb5kdc[27674](info): set up 2 sockets
Sep 24 10:04:39 dist-test-slave-jvsj krb5kdc[27674](info): commencing operation
krb5kdc: starting...
W20260924 10:04:40.554622 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:04:40 dist-test-slave-jvsj krb5kdc[27674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:04:40.777633 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42653
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:42653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:41.098029 27687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:41.098371 27687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:41.098491 27687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:41.105755 27687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:41.105872 27687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:41.105919 27687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:41.105966 27687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:41.106019 27687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:41.116197 27687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:42653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:42653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:41.117573 27687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:41.118947 27687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:41.131191 27695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:41.131331 27692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:41.131270 27693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:41.131819 27687 server_base.cc:1034] running on GCE node
I20260924 10:04:41.132660 27687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:41.134111 27687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:41.135318 27687 hybrid_clock.cc:648] HybridClock initialized: now 1790244281135263 us; error 41 us; skew 500 ppm
I20260924 10:04:41.135666 27687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:41 dist-test-slave-jvsj krb5kdc[27674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:41.138847 27687 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:41.140431 27687 webserver.cc:460] Webserver started at http://127.13.162.62:40471/ using document root <none> and password file <none>
I20260924 10:04:41.141179 27687 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:41.141288 27687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:41.141598 27687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:41.144721 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "728b4dc2d14940d481140c4249074913"
format_stamp: "Formatted at 2026-09-24 10:04:41 on dist-test-slave-jvsj"
server_key: "b3ff1a7af4afe42100241fa6627ff84c"
server_key_iv: "3a404714118325776c2b2a51ed96adcf"
server_key_version: "encryptionkey@0"
I20260924 10:04:41.145485 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "728b4dc2d14940d481140c4249074913"
format_stamp: "Formatted at 2026-09-24 10:04:41 on dist-test-slave-jvsj"
server_key: "b3ff1a7af4afe42100241fa6627ff84c"
server_key_iv: "3a404714118325776c2b2a51ed96adcf"
server_key_version: "encryptionkey@0"
I20260924 10:04:41.150940 27687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:04:41.154572 27702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:41.156021 27687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:04:41.156204 27687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "728b4dc2d14940d481140c4249074913"
format_stamp: "Formatted at 2026-09-24 10:04:41 on dist-test-slave-jvsj"
server_key: "b3ff1a7af4afe42100241fa6627ff84c"
server_key_iv: "3a404714118325776c2b2a51ed96adcf"
server_key_version: "encryptionkey@0"
I20260924 10:04:41.156409 27687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:41.203807 27687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:41.208429 27687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:41.208706 27687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:41.226554 27687 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:42653
I20260924 10:04:41.226557 27753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:42653 every 8 connection(s)
I20260924 10:04:41.227980 27687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:41.229036 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27687
I20260924 10:04:41.229271 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:04:41.229900 13960 external_mini_cluster.cc:1428] Setting key 99d53050de85ce0b2a0e358c4855d266
I20260924 10:04:41.232656 27754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:04:41 dist-test-slave-jvsj krb5kdc[27674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:41.245954 27754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913: Bootstrap starting.
I20260924 10:04:41.250214 27754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:41.251418 27754 log.cc:824] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:41.254354 27754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913: No bootstrap required, opened a new log
I20260924 10:04:41.254522 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:41.231899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.13.162.62:42653)
0924 10:04:41.232696 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:41.232711 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:41.232766 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:04:41.235103 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:41.235132 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:41.235649 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:41.236130 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:41.237476 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:41.239568 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:41.240923 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:41.241966 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:41.242153 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:41.246468 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:41.246481 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:41.246504 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:41.246563 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:41.250280 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:41.251059 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:41.251069 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:41.251078 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:41.251307 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:41.251670 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:41.251674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:41.251676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:41.252257 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:41.252552 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:41.253061 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 10:04:41.253475 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":336,"threads_started":1}
I20260924 10:04:41.258430 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:04:41.262988 27754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728b4dc2d14940d481140c4249074913" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42653 } }
I20260924 10:04:41.263360 27754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:41.263453 27754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728b4dc2d14940d481140c4249074913, State: Initialized, Role: FOLLOWER
I20260924 10:04:41.263993 27754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728b4dc2d14940d481140c4249074913" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42653 } }
I20260924 10:04:41.264194 27754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:04:41.264309 27754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:04:41.264458 27754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:41.266595 27754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728b4dc2d14940d481140c4249074913" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42653 } }
I20260924 10:04:41.267066 27754 leader_election.cc:302] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 728b4dc2d14940d481140c4249074913; no voters:
I20260924 10:04:41.267622 27754 leader_election.cc:288] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:04:41.267750 27759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:04:41.268345 27759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [term 1 LEADER]: Becoming Leader. State: Replica: 728b4dc2d14940d481140c4249074913, State: Running, Role: LEADER
I20260924 10:04:41.268868 27759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728b4dc2d14940d481140c4249074913" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42653 } }
I20260924 10:04:41.269629 27754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:41.271440 27761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 728b4dc2d14940d481140c4249074913. Latest consensus state: current_term: 1 leader_uuid: "728b4dc2d14940d481140c4249074913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728b4dc2d14940d481140c4249074913" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42653 } } }
I20260924 10:04:41.271827 27761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:41.272284 27760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "728b4dc2d14940d481140c4249074913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728b4dc2d14940d481140c4249074913" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 42653 } } }
I20260924 10:04:41.272583 27760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:41.273473 27766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:41.278434 27766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:41.286976 27766 catalog_manager.cc:1347] Generated new cluster ID: 4cb54f22f36c4149bed6381f95993797
I20260924 10:04:41.287106 27766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:04:41.300246 27766 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:04:41.302218 27766 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:04:41.310391 27766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 728b4dc2d14940d481140c4249074913: Generated new TSK 0
I20260924 10:04:41.311491 27766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:04:41.369445 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27687
I20260924 10:04:41.370615 27749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:41.466749 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27687
2026-09-24T10:04:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1933 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:04:41 dist-test-slave-jvsj krb5kdc[27782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:04:41 dist-test-slave-jvsj krb5kdc[27782](info): set up 2 sockets
Sep 24 10:04:41 dist-test-slave-jvsj krb5kdc[27782](info): commencing operation
krb5kdc: starting...
W20260924 10:04:42.478830 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:04:42 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:04:42.697695 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37281
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:43.021266 27795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:43.021548 27795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:43.021667 27795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:43.028895 27795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:43.029004 27795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:43.029062 27795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:43.029139 27795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:43.029194 27795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:43.039295 27795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:43.040769 27795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:43.042114 27795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:43.054031 27800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:43.053967 27801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:43.053992 27803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:43.054697 27795 server_base.cc:1034] running on GCE node
I20260924 10:04:43.055519 27795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:43.056960 27795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:43.058193 27795 hybrid_clock.cc:648] HybridClock initialized: now 1790244283058125 us; error 51 us; skew 500 ppm
I20260924 10:04:43.058576 27795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:43 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244283, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:43.061635 27795 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:43.063119 27795 webserver.cc:460] Webserver started at http://127.13.162.62:41341/ using document root <none> and password file <none>
I20260924 10:04:43.063773 27795 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:43.063887 27795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:43.064168 27795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:43.067206 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "76341fcaaa1a4866b4b1bd9ea5209006"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "f2dc4206a1fc08e0866b1e5887ec0258"
server_key_iv: "9eb31068ef475a6f4d59df3331506087"
server_key_version: "encryptionkey@0"
I20260924 10:04:43.067937 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "76341fcaaa1a4866b4b1bd9ea5209006"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "f2dc4206a1fc08e0866b1e5887ec0258"
server_key_iv: "9eb31068ef475a6f4d59df3331506087"
server_key_version: "encryptionkey@0"
I20260924 10:04:43.073006 27795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:04:43.076530 27810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:43.077956 27795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:04:43.078117 27795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "76341fcaaa1a4866b4b1bd9ea5209006"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "f2dc4206a1fc08e0866b1e5887ec0258"
server_key_iv: "9eb31068ef475a6f4d59df3331506087"
server_key_version: "encryptionkey@0"
I20260924 10:04:43.078297 27795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:43.102483 27795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:43.106633 27795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:43.106887 27795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:43.123549 27795 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:37281
I20260924 10:04:43.123569 27861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:37281 every 8 connection(s)
I20260924 10:04:43.124996 27795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:43.128363 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27795
I20260924 10:04:43.128590 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:04:43.129235 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:43.129361 13960 external_mini_cluster.cc:1428] Setting key d8f6682c8bd622caac413472adc62872
I20260924 10:04:43.143697 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap starting.
Sep 24 10:04:43 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:43.146740 27862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:43.147782 27862 log.cc:824] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:43.150458 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: No bootstrap required, opened a new log
I20260924 10:04:43.151893 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:43.131494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.13.162.62:37281)
0924 10:04:43.132274 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:43.132286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:43.132327 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:04:43.134143 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:43.134164 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:43.134611 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:43.134958 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:43.136604 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:43.137923 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:43.139647 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:43.140353 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:43.140493 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:43.145072 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:43.145081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:43.145094 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:43.145129 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:43.148136 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:43.148791 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:43.148798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:43.148804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:43.148966 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:43.149316 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:43.149319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:43.149322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:43.149811 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:43.150124 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:43.150541 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 10:04:43.150946 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":355,"threads_started":1}
I20260924 10:04:43.158795 27862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:43.159189 27862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:43.159276 27862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76341fcaaa1a4866b4b1bd9ea5209006, State: Initialized, Role: FOLLOWER
I20260924 10:04:43.159862 27862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:43.160084 27862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:04:43.160208 27862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:04:43.160372 27862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:43.162585 27862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:43.163090 27862 leader_election.cc:302] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76341fcaaa1a4866b4b1bd9ea5209006; no voters:
I20260924 10:04:43.163782 27862 leader_election.cc:288] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:04:43.164019 27867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:04:43.165033 27867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 LEADER]: Becoming Leader. State: Replica: 76341fcaaa1a4866b4b1bd9ea5209006, State: Running, Role: LEADER
I20260924 10:04:43.165571 27867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:43.165997 27862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:43.168503 27868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } } }
I20260924 10:04:43.168598 27869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76341fcaaa1a4866b4b1bd9ea5209006. Latest consensus state: current_term: 1 leader_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } } }
I20260924 10:04:43.168872 27868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:43.169025 27869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:43.170644 27875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:43.175172 27875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:43.183437 27875 catalog_manager.cc:1347] Generated new cluster ID: bf0c14dbbf6848f884e605b500904380
I20260924 10:04:43.183560 27875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:04:43.209458 27875 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:04:43.211440 27875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:04:43.221238 27875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Generated new TSK 0
I20260924 10:04:43.222297 27875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:04:43.288606 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37281
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:43.616892 27888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:43.617187 27888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:43.617307 27888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:43.624581 27888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:43.624696 27888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:43.624845 27888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:04:43.634685 27888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:43.636039 27888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:43.637324 27888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:43.650664 27896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:43.651260 27894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:43.650597 27893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:43.651782 27888 server_base.cc:1034] running on GCE node
I20260924 10:04:43.652426 27888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:43.653403 27888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:43.654611 27888 hybrid_clock.cc:648] HybridClock initialized: now 1790244283654561 us; error 37 us; skew 500 ppm
I20260924 10:04:43.654947 27888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:43 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244283, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:43.658077 27888 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:04:43.659646 27888 webserver.cc:460] Webserver started at http://127.13.162.1:40569/ using document root <none> and password file <none>
I20260924 10:04:43.660338 27888 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:43.660457 27888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:43.660740 27888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:43.663723 27888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "9e668593e7d34ba4b8423cfd4c87df42"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "caf8e9b621afefae89d9e305f9bdb3f0"
server_key_iv: "5f0f6a5f558e4728f9e10ae26a2852d0"
server_key_version: "encryptionkey@0"
I20260924 10:04:43.664516 27888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "9e668593e7d34ba4b8423cfd4c87df42"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "caf8e9b621afefae89d9e305f9bdb3f0"
server_key_iv: "5f0f6a5f558e4728f9e10ae26a2852d0"
server_key_version: "encryptionkey@0"
I20260924 10:04:43.669689 27888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:04:43.673394 27903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:43.674867 27888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:04:43.675035 27888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "9e668593e7d34ba4b8423cfd4c87df42"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "caf8e9b621afefae89d9e305f9bdb3f0"
server_key_iv: "5f0f6a5f558e4728f9e10ae26a2852d0"
server_key_version: "encryptionkey@0"
I20260924 10:04:43.675235 27888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:43.697288 27888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:43.701280 27888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:43.701547 27888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:43.702536 27888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:43.703898 27888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:43.703980 27888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:43.704092 27888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:43.704142 27888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:43.734436 27888 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41995
I20260924 10:04:43.734481 28015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41995 every 8 connection(s)
I20260924 10:04:43.735725 27888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:04:43.742825 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 27888
I20260924 10:04:43.743024 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:04:43.743759 13960 external_mini_cluster.cc:1428] Setting key e0d2c39c0b85c584a3f3c92fd39799da
Sep 24 10:04:43 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244283, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:43.756392 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:43.738182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:58201 (local address 127.13.162.62:37281)
0924 10:04:43.738681 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:43.738687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:43.739637 (+ 950us) server_negotiation.cc:407] Connection header received
0924 10:04:43.741428 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:43.741432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:43.741508 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:43.741602 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:43.743264 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:43.744308 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:43.746403 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:43.746963 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:43.747083 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:43.750822 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:43.750829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:43.750832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:43.750872 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:43.753284 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:43.754041 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:43.754048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:43.754052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:43.754191 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:43.754620 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:43.754624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:43.754627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:43.754950 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:43.755094 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:43.755810 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 10:04:43.756031 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260924 10:04:43.757181 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:43.738793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:58201)
0924 10:04:43.739480 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:43.739524 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:43.741174 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:43.741816 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:43.741825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:43.742451 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:43.743073 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:43.743094 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:43.744503 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:43.744509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:43.746240 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:43.746252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:43.747182 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:43.747187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:43.747395 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:43.748071 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:43.748100 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:43.750571 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:43.753467 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:43.753475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:43.753488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:43.753922 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:43.754283 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:43.754286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:43.754288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:43.754522 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:43.755055 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:43.755061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:43.755398 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 10:04:43.755657 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:04:43.756070 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":259,"threads_started":1}
I20260924 10:04:43.758775 28016 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37281
I20260924 10:04:43.759078 28016 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:43.759824 28016 heartbeater.cc:511] Master 127.13.162.62:37281 requested a full tablet report, sending...
I20260924 10:04:43.762071 27827 ts_manager.cc:194] Registered new tserver with Master: 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1:41995)
I20260924 10:04:43.763634 27827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:58201
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:04:43.791607 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37281
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:44.111474 28022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:44.111786 28022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:44.111936 28022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:44.119233 28022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:44.119374 28022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:44.119527 28022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:04:44.130093 28022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:44.131460 28022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:44.132745 28022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:44.145223 28028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:44.145347 28027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:44.145422 28030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:44.146364 28022 server_base.cc:1034] running on GCE node
I20260924 10:04:44.146981 28022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:44.147832 28022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:44.149037 28022 hybrid_clock.cc:648] HybridClock initialized: now 1790244284148990 us; error 37 us; skew 500 ppm
I20260924 10:04:44.149333 28022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:44 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:44.152295 28022 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:04:44.153700 28022 webserver.cc:460] Webserver started at http://127.13.162.2:46637/ using document root <none> and password file <none>
I20260924 10:04:44.154429 28022 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:44.154549 28022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:44.154821 28022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:44.157706 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "8224135850df46ecafbde4fbb31da610"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "eb2ac725946704987da85faac7b2ff06"
server_key_iv: "f7eaf65fd4fd3f7c62480bf906c8ec88"
server_key_version: "encryptionkey@0"
I20260924 10:04:44.158507 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "8224135850df46ecafbde4fbb31da610"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "eb2ac725946704987da85faac7b2ff06"
server_key_iv: "f7eaf65fd4fd3f7c62480bf906c8ec88"
server_key_version: "encryptionkey@0"
I20260924 10:04:44.163823 28022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:04:44.167712 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:44.169396 28022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:04:44.169574 28022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "8224135850df46ecafbde4fbb31da610"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "eb2ac725946704987da85faac7b2ff06"
server_key_iv: "f7eaf65fd4fd3f7c62480bf906c8ec88"
server_key_version: "encryptionkey@0"
I20260924 10:04:44.169766 28022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:44.184553 28022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:44.188890 28022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:44.189168 28022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:44.190152 28022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:44.191672 28022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:44.191759 28022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:44.191883 28022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:44.191938 28022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:44.222023 28022 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:33773
I20260924 10:04:44.222038 28149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:33773 every 8 connection(s)
I20260924 10:04:44.223270 28022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:04:44.232460 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28022
I20260924 10:04:44.232658 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:04:44.233135 13960 external_mini_cluster.cc:1428] Setting key c100ed0fbe4d2eb257827580ed98d52c
Sep 24 10:04:44 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:44.242383 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.225645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:50757 (local address 127.13.162.62:37281)
0924 10:04:44.225823 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:44.225828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:44.227029 (+ 1201us) server_negotiation.cc:407] Connection header received
0924 10:04:44.228749 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:44.228754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:44.228842 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:44.228927 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:44.230401 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.231111 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.232855 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.233291 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.233392 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.236966 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:44.236972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:44.236975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:44.237008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:44.239447 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:44.240141 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:44.240145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:44.240147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:44.240278 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:44.240686 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:44.240690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:44.240692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:44.240968 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:44.241096 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:44.241793 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 10:04:44.242024 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:04:44.243424 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.226129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:50757)
0924 10:04:44.226876 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:44.226914 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:44.228487 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:44.229168 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:44.229175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:44.229669 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:44.230230 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.230248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.231254 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.231258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.232718 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.232733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.233514 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.233521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.233781 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.234470 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:44.234493 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:44.236747 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:44.239576 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:44.239586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:44.239604 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:44.240021 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:44.240378 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:44.240383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:44.240387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:44.240577 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:44.241090 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:44.241098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:44.241440 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 10:04:44.241672 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 10:04:44.242280 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":268,"threads_started":1}
I20260924 10:04:44.244946 28150 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37281
I20260924 10:04:44.245244 28150 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:44.246099 28150 heartbeater.cc:511] Master 127.13.162.62:37281 requested a full tablet report, sending...
I20260924 10:04:44.247682 27827 ts_manager.cc:194] Registered new tserver with Master: 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773)
I20260924 10:04:44.248811 27827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:50757
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:04:44.280037 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37281
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:44.613539 28155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:44.613870 28155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:44.614022 28155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:44.621322 28155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:44.621459 28155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:44.621618 28155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:04:44.631624 28155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:44.632972 28155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:44.634373 28155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:44.647755 28160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:44.647802 28161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:44.647828 28163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:44.648953 28155 server_base.cc:1034] running on GCE node
I20260924 10:04:44.649569 28155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:44.650610 28155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:44.651880 28155 hybrid_clock.cc:648] HybridClock initialized: now 1790244284651799 us; error 70 us; skew 500 ppm
I20260924 10:04:44.652204 28155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:44 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:44.660377 28155 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:04:44.662079 28155 webserver.cc:460] Webserver started at http://127.13.162.3:42265/ using document root <none> and password file <none>
I20260924 10:04:44.662808 28155 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:44.662930 28155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:44.663239 28155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:04:44.666277 28155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "5affdd7815fc4668a2d73d056683f098"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "314a6006d50c3f0feb4d91ff7eb28634"
server_key_iv: "de475da33ceb51ffaf318ad97d471181"
server_key_version: "encryptionkey@0"
I20260924 10:04:44.667122 28155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "5affdd7815fc4668a2d73d056683f098"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "314a6006d50c3f0feb4d91ff7eb28634"
server_key_iv: "de475da33ceb51ffaf318ad97d471181"
server_key_version: "encryptionkey@0"
I20260924 10:04:44.672459 28155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:04:44.676312 28170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:44.677829 28155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:04:44.678145 28155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "5affdd7815fc4668a2d73d056683f098"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "314a6006d50c3f0feb4d91ff7eb28634"
server_key_iv: "de475da33ceb51ffaf318ad97d471181"
server_key_version: "encryptionkey@0"
I20260924 10:04:44.678413 28155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:44.694303 28155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:44.698753 28155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:44.699040 28155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:44.700049 28155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:44.701478 28155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:44.701555 28155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:44.701663 28155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:44.701717 28155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:44.732996 28155 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:41077
I20260924 10:04:44.733034 28282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:41077 every 8 connection(s)
I20260924 10:04:44.734372 28155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:04:44.741662 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28155
I20260924 10:04:44.741849 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:04:44.742412 13960 external_mini_cluster.cc:1428] Setting key 1b604a2cff261525c167bbd55498ac1e
Sep 24 10:04:44 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:44.755251 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.736967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:57743 (local address 127.13.162.62:37281)
0924 10:04:44.737176 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:44.737182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:44.738472 (+ 1290us) server_negotiation.cc:407] Connection header received
0924 10:04:44.740553 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:44.740559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:44.740672 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:44.740769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:44.742781 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.743771 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.745818 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.746244 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.746399 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.749777 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:44.749783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:44.749786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:44.749822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:44.752360 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:44.753105 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:44.753110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:44.753112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:44.753251 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:44.753624 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:44.753627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:44.753629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:44.753944 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:44.754066 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:44.754807 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 10:04:44.755004 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:04:44.756356 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.737445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:57743)
0924 10:04:44.738255 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:44.738301 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:44.740238 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:44.741035 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:44.741044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:44.741710 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:44.742560 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.742590 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.743979 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.743984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.745675 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.745693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.746468 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.746474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.746686 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.747357 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:44.747380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:44.749546 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:44.752510 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:44.752517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:44.752530 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:44.752944 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:44.753336 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:44.753339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:44.753341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:44.753533 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:44.754042 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:44.754048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:44.754439 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 10:04:44.754670 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 10:04:44.755093 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":309,"threads_started":1}
I20260924 10:04:44.757844 28283 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37281
I20260924 10:04:44.758248 28283 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:44.759081 28283 heartbeater.cc:511] Master 127.13.162.62:37281 requested a full tablet report, sending...
I20260924 10:04:44.760560 27827 ts_manager.cc:194] Registered new tserver with Master: 5affdd7815fc4668a2d73d056683f098 (127.13.162.3:41077)
I20260924 10:04:44.761363 27827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:57743
I20260924 10:04:44.767009 28016 heartbeater.cc:503] Master 127.13.162.62:37281 was elected leader, sending a full tablet report...
I20260924 10:04:44.770910 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:04:44.787794 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.775221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.13.162.62:37281)
0924 10:04:44.775395 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:44.775401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:44.775783 (+ 382us) server_negotiation.cc:407] Connection header received
0924 10:04:44.775947 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:44.775951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:44.776021 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:44.776106 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:44.777410 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.778171 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.779714 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.780119 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.780228 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.782582 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:44.782587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:44.782590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:44.782623 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:44.785249 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:44.785859 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:44.785864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:44.785866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:44.785998 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:44.786359 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:44.786363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:44.786365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:44.786691 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:44.786939 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:44.787249 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 10:04:44.787460 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:04:44.791889 27827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:04:44.798854 27827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:04:44.836009 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.812263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.13.162.3:41077)
0924 10:04:44.828204 (+ 15941us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:44.828214 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:44.828248 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:04:44.828350 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:44.828358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:44.828621 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:44.828809 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:44.829781 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.830963 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.833735 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.834858 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.834983 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.835235 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:44.835509 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 10:04:44.835668 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15778,"thread_start_us":15646,"threads_started":1}
I20260924 10:04:44.836218 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.812029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41077 (local address 127.0.0.1:41934)
0924 10:04:44.827769 (+ 15740us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:44.827795 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:44.827917 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:44.829105 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:44.829110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:44.829138 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:44.829553 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.829563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.831096 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.831100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.833554 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.833569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.835050 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.835057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.835265 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.835312 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:04:44.835442 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:04:44.835648 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15609,"mutex_wait_us":9,"thread_start_us":9156,"threads_started":1}
I20260924 10:04:44.837170 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.814740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.13.162.2:33773)
0924 10:04:44.828535 (+ 13795us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:44.828543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:44.828787 (+ 244us) server_negotiation.cc:407] Connection header received
0924 10:04:44.828895 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:44.828903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:44.829120 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:44.829312 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:44.830129 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.831622 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.833982 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.835530 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.835664 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.835867 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:44.836696 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 10:04:44.836892 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13647,"thread_start_us":13526,"threads_started":1}
I20260924 10:04:44.837909 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.811764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38224)
0924 10:04:44.821609 (+ 9845us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:44.821657 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:44.821865 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:44.828721 (+ 6856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:44.828729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:44.828766 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:44.829312 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.829329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.831496 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.831501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.833860 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.833870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.837020 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.837026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.837190 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.837220 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:04:44.837307 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:04:44.837459 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9562,"mutex_wait_us":204,"thread_start_us":16720,"threads_started":1}
I20260924 10:04:44.838099 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.811914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.13.162.1:41995)
0924 10:04:44.827092 (+ 15178us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:44.827103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:44.827140 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:04:44.827260 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:44.827271 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:44.827586 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:44.829528 (+ 1942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:44.829801 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.831360 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.834026 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.835530 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.836664 (+ 1134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.836889 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:44.837624 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 10:04:44.837805 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14965,"thread_start_us":14799,"threads_started":1}
I20260924 10:04:44.837937 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:44.814519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:33773 (local address 127.0.0.1:46238)
0924 10:04:44.828507 (+ 13988us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:44.828528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:44.828658 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:44.829544 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:44.829549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:44.829571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:44.829961 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.829972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.831775 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.831780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.833783 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:44.833797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:44.836035 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:44.836042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:44.836207 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:44.836237 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:04:44.836319 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:04:44.837451 (+ 1132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13863,"mutex_wait_us":26,"thread_start_us":13471,"threads_started":1}
I20260924 10:04:44.840438 28218 tablet_service.cc:1467] Processing CreateTablet for tablet 46d2ab78e56f4ac299b6f1764624a4e3 (DEFAULT_TABLE table=test-table [id=6a73887a4ac141959d3b0a6e063a7e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:44.841944 28218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d2ab78e56f4ac299b6f1764624a4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:44.841921 28085 tablet_service.cc:1467] Processing CreateTablet for tablet 46d2ab78e56f4ac299b6f1764624a4e3 (DEFAULT_TABLE table=test-table [id=6a73887a4ac141959d3b0a6e063a7e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:44.841929 27951 tablet_service.cc:1467] Processing CreateTablet for tablet 46d2ab78e56f4ac299b6f1764624a4e3 (DEFAULT_TABLE table=test-table [id=6a73887a4ac141959d3b0a6e063a7e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:44.843439 27951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d2ab78e56f4ac299b6f1764624a4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:44.843863 28085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d2ab78e56f4ac299b6f1764624a4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:44.855894 28301 tablet_bootstrap.cc:492] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Bootstrap starting.
I20260924 10:04:44.858402 28303 tablet_bootstrap.cc:492] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Bootstrap starting.
I20260924 10:04:44.859143 28301 tablet_bootstrap.cc:654] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:44.859185 28302 tablet_bootstrap.cc:492] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Bootstrap starting.
I20260924 10:04:44.860488 28301 log.cc:824] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:44.862767 28302 tablet_bootstrap.cc:654] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:44.862769 28303 tablet_bootstrap.cc:654] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:44.863785 28302 log.cc:824] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:44.863978 28303 log.cc:824] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:44.864482 28301 tablet_bootstrap.cc:492] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: No bootstrap required, opened a new log
I20260924 10:04:44.864776 28301 ts_tablet_manager.cc:1397] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:04:44.866739 28303 tablet_bootstrap.cc:492] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: No bootstrap required, opened a new log
I20260924 10:04:44.866930 28302 tablet_bootstrap.cc:492] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: No bootstrap required, opened a new log
I20260924 10:04:44.867012 28303 ts_tablet_manager.cc:1397] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 10:04:44.867182 28302 ts_tablet_manager.cc:1397] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:04:44.875032 28303 raft_consensus.cc:348] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:44.875432 28303 raft_consensus.cc:374] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:44.875538 28303 raft_consensus.cc:729] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8224135850df46ecafbde4fbb31da610, State: Initialized, Role: FOLLOWER
I20260924 10:04:44.875620 28301 raft_consensus.cc:348] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:44.876072 28301 raft_consensus.cc:374] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:44.876168 28301 raft_consensus.cc:729] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5affdd7815fc4668a2d73d056683f098, State: Initialized, Role: FOLLOWER
I20260924 10:04:44.876183 28303 consensus_queue.cc:260] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:44.876847 28301 consensus_queue.cc:260] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:44.877348 28150 heartbeater.cc:503] Master 127.13.162.62:37281 was elected leader, sending a full tablet report...
I20260924 10:04:44.877302 28302 raft_consensus.cc:348] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:44.877753 28302 raft_consensus.cc:374] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:44.877832 28302 raft_consensus.cc:729] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e668593e7d34ba4b8423cfd4c87df42, State: Initialized, Role: FOLLOWER
I20260924 10:04:44.878121 28303 ts_tablet_manager.cc:1428] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260924 10:04:44.878588 28302 consensus_queue.cc:260] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:44.879122 28283 heartbeater.cc:503] Master 127.13.162.62:37281 was elected leader, sending a full tablet report...
I20260924 10:04:44.879817 28301 ts_tablet_manager.cc:1428] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260924 10:04:44.881850 28302 ts_tablet_manager.cc:1428] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260924 10:04:44.975152 28151 tablet.cc:2369] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:04:44.986239 28284 tablet.cc:2369] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:04:44.987902 28017 tablet.cc:2369] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:04:45.014214 28309 raft_consensus.cc:482] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:04:45.014612 28309 raft_consensus.cc:504] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:45.016860 28309 leader_election.cc:288] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5affdd7815fc4668a2d73d056683f098 (127.13.162.3:41077), 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773)
I20260924 10:04:45.024755 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.017465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:54745 (local address 127.13.162.3:41077)
0924 10:04:45.017681 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:45.017688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:45.018304 (+ 616us) server_negotiation.cc:407] Connection header received
0924 10:04:45.018667 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:45.018673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:45.018752 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:45.018839 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:45.019561 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.020693 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.023034 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.023803 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.023959 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.024037 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:45.024364 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:04:45.024500 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:04:45.025013 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.017926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:55539 (local address 127.13.162.2:33773)
0924 10:04:45.018123 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:45.018129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:45.018637 (+ 508us) server_negotiation.cc:407] Connection header received
0924 10:04:45.018812 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:45.018818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:45.018892 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:45.018978 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:45.019599 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.020322 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.022854 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.024079 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.024242 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.024318 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:45.024607 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 10:04:45.024749 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:04:45.025055 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.017336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:33773 (local address 127.13.162.1:55539)
0924 10:04:45.018531 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:45.018546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:45.018679 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:45.019021 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:45.019026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:45.019061 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:45.019425 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:45.019434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.020462 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.020467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:45.022648 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:45.022665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.024291 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.024298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:45.024463 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.024492 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:04:45.024575 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:04:45.024703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":75,"thread_start_us":522,"threads_started":1}
I20260924 10:04:45.025821 28311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.017239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41077 (local address 127.13.162.1:54745)
0924 10:04:45.018167 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:45.018189 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:45.018479 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:45.018888 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:45.018893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:45.018915 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:45.019399 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:45.019416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.020836 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.020841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:45.022895 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:45.022906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.024005 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.024011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:45.024166 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.024198 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:04:45.025369 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0924 10:04:45.025493 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"spinlock_wait_cycles":29056,"thread_start_us":953,"threads_started":1}
I20260924 10:04:45.026132 28105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d2ab78e56f4ac299b6f1764624a4e3" candidate_uuid: "9e668593e7d34ba4b8423cfd4c87df42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8224135850df46ecafbde4fbb31da610" is_pre_election: true
I20260924 10:04:45.026738 28105 raft_consensus.cc:2454] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e668593e7d34ba4b8423cfd4c87df42 in term 0.
I20260924 10:04:45.026891 28238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d2ab78e56f4ac299b6f1764624a4e3" candidate_uuid: "9e668593e7d34ba4b8423cfd4c87df42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5affdd7815fc4668a2d73d056683f098" is_pre_election: true
I20260924 10:04:45.027419 28238 raft_consensus.cc:2454] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e668593e7d34ba4b8423cfd4c87df42 in term 0.
I20260924 10:04:45.027518 27907 leader_election.cc:302] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8224135850df46ecafbde4fbb31da610, 9e668593e7d34ba4b8423cfd4c87df42; no voters:
I20260924 10:04:45.027969 28309 raft_consensus.cc:2781] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:04:45.028111 28309 raft_consensus.cc:482] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:04:45.028208 28309 raft_consensus.cc:3037] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:45.030495 28309 raft_consensus.cc:504] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:45.031129 28309 leader_election.cc:288] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [CANDIDATE]: Term 1 election: Requested vote from peers 5affdd7815fc4668a2d73d056683f098 (127.13.162.3:41077), 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773)
I20260924 10:04:45.031478 28238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d2ab78e56f4ac299b6f1764624a4e3" candidate_uuid: "9e668593e7d34ba4b8423cfd4c87df42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5affdd7815fc4668a2d73d056683f098"
I20260924 10:04:45.031692 28238 raft_consensus.cc:3037] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:45.031662 28105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d2ab78e56f4ac299b6f1764624a4e3" candidate_uuid: "9e668593e7d34ba4b8423cfd4c87df42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8224135850df46ecafbde4fbb31da610"
I20260924 10:04:45.031859 28105 raft_consensus.cc:3037] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:45.034912 28238 raft_consensus.cc:2454] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e668593e7d34ba4b8423cfd4c87df42 in term 1.
I20260924 10:04:45.034909 28105 raft_consensus.cc:2454] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e668593e7d34ba4b8423cfd4c87df42 in term 1.
I20260924 10:04:45.035418 27904 leader_election.cc:302] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5affdd7815fc4668a2d73d056683f098, 9e668593e7d34ba4b8423cfd4c87df42; no voters:
I20260924 10:04:45.035773 28309 raft_consensus.cc:2781] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:04:45.036262 28309 raft_consensus.cc:686] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 LEADER]: Becoming Leader. State: Replica: 9e668593e7d34ba4b8423cfd4c87df42, State: Running, Role: LEADER
I20260924 10:04:45.037091 28309 consensus_queue.cc:237] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } }
I20260924 10:04:45.043223 27825 catalog_manager.cc:5495] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1). New cstate: current_term: 1 leader_uuid: "9e668593e7d34ba4b8423cfd4c87df42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } health_report { overall_health: UNKNOWN } } }
I20260924 10:04:45.087822 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.082037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.13.162.1:41995)
0924 10:04:45.082227 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:45.082232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:45.082250 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:04:45.082471 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:45.082476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:45.082562 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:45.082698 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:04:45.083101 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.084041 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.085477 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.085919 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.086045 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.086670 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:04:45.087207 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:04:45.087355 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:45.087470 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 10:04:45.087562 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:04:45.096253 28238 raft_consensus.cc:1261] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Refusing update from remote peer 9e668593e7d34ba4b8423cfd4c87df42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:45.096396 28105 raft_consensus.cc:1261] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Refusing update from remote peer 9e668593e7d34ba4b8423cfd4c87df42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:45.097234 28309 consensus_queue.cc:1035] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:45.097527 28312 consensus_queue.cc:1035] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:45.110422 28315 mvcc.cc:205] Tried to move back new op lower bound from 7332840591743537152 to 7332840591517106176. Current Snapshot: MvccSnapshot[applied={T|T < 7332840591743537152}]
I20260924 10:04:45.124542 28317 mvcc.cc:205] Tried to move back new op lower bound from 7332840591743537152 to 7332840591517106176. Current Snapshot: MvccSnapshot[applied={T|T < 7332840591743537152}]
I20260924 10:04:45.143680 27827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:04:45.144090 27827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:04:45.148492 27827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Sending DeleteTablet for 3 replicas of tablet 46d2ab78e56f4ac299b6f1764624a4e3
I20260924 10:04:45.149698 27951 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d2ab78e56f4ac299b6f1764624a4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:45 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:38224
I20260924 10:04:45.149709 28218 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d2ab78e56f4ac299b6f1764624a4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:45 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:41934
I20260924 10:04:45.151062 28085 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d2ab78e56f4ac299b6f1764624a4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:45 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:46238
I20260924 10:04:45.158149 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27888
I20260924 10:04:45.158872 28325 tablet_replica.cc:330] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: stopping tablet replica
I20260924 10:04:45.159389 28325 raft_consensus.cc:2229] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:04:45.160071 28325 raft_consensus.cc:2258] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:45.174518 28325 ts_tablet_manager.cc:1905] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:45.181360 28327 tablet_replica.cc:330] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: stopping tablet replica
I20260924 10:04:45.181869 28327 raft_consensus.cc:2229] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:04:45.182459 28327 raft_consensus.cc:2258] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 10:04:45 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:04:45.184898 28327 ts_tablet_manager.cc:1905] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:45.192566 28327 ts_tablet_manager.cc:1918] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:45.192749 28327 log.cc:1196] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/46d2ab78e56f4ac299b6f1764624a4e3
I20260924 10:04:45.193295 28327 ts_tablet_manager.cc:1939] T 46d2ab78e56f4ac299b6f1764624a4e3 P 8224135850df46ecafbde4fbb31da610: Deleting consensus metadata
I20260924 10:04:45.193590 28325 ts_tablet_manager.cc:1918] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:45.193804 28325 log.cc:1196] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/46d2ab78e56f4ac299b6f1764624a4e3
I20260924 10:04:45.194514 28325 ts_tablet_manager.cc:1939] T 46d2ab78e56f4ac299b6f1764624a4e3 P 9e668593e7d34ba4b8423cfd4c87df42: Deleting consensus metadata
I20260924 10:04:45.195700 27814 catalog_manager.cc:4841] TS 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773): tablet 46d2ab78e56f4ac299b6f1764624a4e3 (table test-table [id=6a73887a4ac141959d3b0a6e063a7e5c]) successfully deleted
I20260924 10:04:45.196327 27813 catalog_manager.cc:4841] TS 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1:41995): tablet 46d2ab78e56f4ac299b6f1764624a4e3 (table test-table [id=6a73887a4ac141959d3b0a6e063a7e5c]) successfully deleted
I20260924 10:04:45.196525 28326 tablet_replica.cc:330] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: stopping tablet replica
I20260924 10:04:45.197047 28326 raft_consensus.cc:2229] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:04:45.197628 28326 raft_consensus.cc:2258] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:45.201409 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.159419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.13.162.1:41995)
0924 10:04:45.159600 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:45.159606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:45.171824 (+ 12218us) server_negotiation.cc:407] Connection header received
0924 10:04:45.172405 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:45.172412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:45.172533 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:45.172615 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:45.175067 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.176594 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.178808 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.179554 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.179683 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.185820 (+ 6137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:45.185839 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:45.185858 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:45.185922 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:45.190471 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:45.198186 (+ 7715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:45.198197 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:45.198201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:45.198527 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:45.199175 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:45.199181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:45.199184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:45.199670 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:45.200017 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:45.200700 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 10:04:45.201002 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":223,"server-negotiator.queue_time_us":65}
I20260924 10:04:45.201400 28326 ts_tablet_manager.cc:1905] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:45.203393 28011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:45.209365 28326 ts_tablet_manager.cc:1918] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:45.209560 28326 log.cc:1196] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/46d2ab78e56f4ac299b6f1764624a4e3
I20260924 10:04:45.209995 28326 ts_tablet_manager.cc:1939] T 46d2ab78e56f4ac299b6f1764624a4e3 P 5affdd7815fc4668a2d73d056683f098: Deleting consensus metadata
I20260924 10:04:45.211730 27811 catalog_manager.cc:4841] TS 5affdd7815fc4668a2d73d056683f098 (127.13.162.3:41077): tablet 46d2ab78e56f4ac299b6f1764624a4e3 (table test-table [id=6a73887a4ac141959d3b0a6e063a7e5c]) successfully deleted
I20260924 10:04:45.345506 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27888
I20260924 10:04:45.360728 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28022
Sep 24 10:04:45 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:04:45.376281 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.361422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.13.162.2:33773)
0924 10:04:45.361579 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:45.361583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:45.361608 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:04:45.361798 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:45.361803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:45.361880 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:45.361967 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:45.363295 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.364118 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.365494 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.366014 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.366116 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.370355 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:45.370379 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:45.370401 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:45.370449 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:45.373320 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:45.374071 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:45.374079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:45.374081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:45.374217 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:45.374761 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:45.374764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:45.374767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:45.375163 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:45.375450 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:45.375757 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 10:04:45.375969 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:04:45.376933 28145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:45.465592 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28022
I20260924 10:04:45.479305 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28155
Sep 24 10:04:45 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:04:45.494912 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:45.480102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.13.162.3:41077)
0924 10:04:45.480284 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:45.480290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:45.480310 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:04:45.480448 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:45.480454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:45.480526 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:45.480613 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:45.481916 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.482762 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.484115 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:45.484523 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:45.484638 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:45.489264 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:45.489282 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:45.489299 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:45.489341 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:45.492015 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:45.492740 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:45.492748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:45.492750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:45.492909 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:45.493306 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:45.493310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:45.493312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:45.493678 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:45.493995 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:45.494300 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:04:45.494554 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:04:45.495584 28278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:45.585716 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28155
I20260924 10:04:45.599444 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 27795
I20260924 10:04:45.600572 27857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:45.709959 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 27795
I20260924 10:04:45.727991 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37281
--webserver_interface=127.13.162.62
--webserver_port=41341
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:46.056181 28333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:46.056509 28333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:46.056645 28333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:46.063958 28333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:46.064092 28333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:46.064177 28333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:46.064248 28333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:46.064306 28333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:46.074443 28333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=41341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:46.075785 28333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:46.077112 28333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:46.099552 28340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:46.099776 28339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:46.099534 28342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:46.100535 28333 server_base.cc:1034] running on GCE node
I20260924 10:04:46.101413 28333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:46.103145 28333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:46.104372 28333 hybrid_clock.cc:648] HybridClock initialized: now 1790244286104296 us; error 55 us; skew 500 ppm
I20260924 10:04:46.104794 28333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:46 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244286, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:46.108352 28333 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:46.110000 28333 webserver.cc:460] Webserver started at http://127.13.162.62:41341/ using document root <none> and password file <none>
I20260924 10:04:46.110819 28333 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:46.110942 28333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:46.117179 28333 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 10:04:46.120270 28349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:46.121786 28333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:04:46.121959 28333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "76341fcaaa1a4866b4b1bd9ea5209006"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "f2dc4206a1fc08e0866b1e5887ec0258"
server_key_iv: "9eb31068ef475a6f4d59df3331506087"
server_key_version: "encryptionkey@0"
I20260924 10:04:46.123102 28333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:46.156061 28333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:46.160816 28333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:46.161108 28333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:46.179386 28333 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:37281
I20260924 10:04:46.179419 28400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:37281 every 8 connection(s)
I20260924 10:04:46.180904 28333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:46.187889 28401 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:04:46.189937 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28333
I20260924 10:04:46.190907 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:41995
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:37281
--webserver_port=40569
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:04:46.191923 28401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap starting.
I20260924 10:04:46.213151 28401 log.cc:824] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:46.225332 28401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:04:46.225826 28401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap complete.
I20260924 10:04:46.235086 28401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:46.235950 28401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76341fcaaa1a4866b4b1bd9ea5209006, State: Initialized, Role: FOLLOWER
I20260924 10:04:46.236593 28401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:46.236820 28401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:04:46.236939 28401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:04:46.237105 28401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:04:46.241125 28401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:46.241701 28401 leader_election.cc:302] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76341fcaaa1a4866b4b1bd9ea5209006; no voters:
I20260924 10:04:46.242354 28401 leader_election.cc:288] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:04:46.242504 28405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:04:46.243692 28405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 LEADER]: Becoming Leader. State: Replica: 76341fcaaa1a4866b4b1bd9ea5209006, State: Running, Role: LEADER
I20260924 10:04:46.244338 28405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:46.244740 28401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:46.246934 28406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } } }
I20260924 10:04:46.247375 28406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:46.249634 28407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76341fcaaa1a4866b4b1bd9ea5209006. Latest consensus state: current_term: 2 leader_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } } }
I20260924 10:04:46.249944 28407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:46.251621 28411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:46.260751 28411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:46.262507 28411 catalog_manager.cc:1259] Loaded cluster ID: bf0c14dbbf6848f884e605b500904380
I20260924 10:04:46.262699 28411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:04:46.266112 28411 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 10:04:46.279856 28422 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:04:46.283330 28411 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 10:04:46.290856 28411 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:04:46.292202 28411 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Loaded TSK: 0
I20260924 10:04:46.293608 28411 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:04:46.571935 28403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:46.572304 28403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:46.572446 28403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:46.579947 28403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:46.580096 28403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:46.580259 28403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:04:46.590258 28403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:41995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=40569
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:46.591663 28403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:46.592976 28403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:46.608343 28431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:46.608297 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:46.608698 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:46.609395 28403 server_base.cc:1034] running on GCE node
I20260924 10:04:46.610071 28403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:46.611158 28403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:46.612361 28403 hybrid_clock.cc:648] HybridClock initialized: now 1790244286612288 us; error 64 us; skew 500 ppm
I20260924 10:04:46.612671 28403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:46 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244286, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:46.615921 28403 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:04:46.617375 28403 webserver.cc:460] Webserver started at http://127.13.162.1:40569/ using document root <none> and password file <none>
I20260924 10:04:46.618049 28403 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:46.618166 28403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:46.623718 28403 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 10:04:46.626693 28438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:46.628278 28403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:04:46.628427 28403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "9e668593e7d34ba4b8423cfd4c87df42"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "caf8e9b621afefae89d9e305f9bdb3f0"
server_key_iv: "5f0f6a5f558e4728f9e10ae26a2852d0"
server_key_version: "encryptionkey@0"
I20260924 10:04:46.629393 28403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:46.664376 28403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:46.668670 28403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:46.668929 28403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:46.669903 28403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:46.671314 28403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:46.671409 28403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:46.671520 28403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:46.671595 28403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:46.702548 28403 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41995
I20260924 10:04:46.702632 28550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41995 every 8 connection(s)
I20260924 10:04:46.703958 28403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:04:46.707597 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28403
I20260924 10:04:46.708312 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:33773
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:37281
--webserver_port=46637
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:04:46 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244286, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:46.728586 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:46.706923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:59163)
0924 10:04:46.707735 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:46.707788 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:46.709541 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:46.711028 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:46.711039 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:46.711643 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:46.712176 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:46.712193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:46.713906 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:46.713911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:46.715366 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:46.715379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:46.716769 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:46.716775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:46.716980 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:46.717640 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:46.717663 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:46.719797 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:46.723945 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:46.723952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:46.723966 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:46.724436 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:46.724980 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:46.724983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:46.724986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:46.725211 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:46.726213 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:46.726219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:46.726639 (+ 420us) client_negotiation.cc:769] Sending connection context
0924 10:04:46.726884 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:04:46.727364 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260924 10:04:46.728974 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:46.706719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:59163 (local address 127.13.162.62:37281)
0924 10:04:46.707316 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:46.707333 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:46.707954 (+ 621us) server_negotiation.cc:407] Connection header received
0924 10:04:46.710093 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:46.710108 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:46.710513 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:46.710863 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:46.712388 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:46.713702 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:46.715514 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:46.716455 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:46.716641 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:46.720098 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:46.720107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:46.720127 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:46.720168 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:46.723763 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:46.724597 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:46.724608 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:46.724617 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:46.724853 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:46.725335 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:46.725341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:46.725344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:46.726070 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:46.726288 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:46.727177 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 10:04:46.727662 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
I20260924 10:04:46.731937 28551 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37281
I20260924 10:04:46.732235 28551 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:46.733098 28551 heartbeater.cc:511] Master 127.13.162.62:37281 requested a full tablet report, sending...
I20260924 10:04:46.735569 28366 ts_manager.cc:194] Registered new tserver with Master: 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1:41995)
I20260924 10:04:46.737761 28366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:59163
W20260924 10:04:47.031842 28555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:47.032109 28555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:47.032199 28555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:47.041070 28555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:47.041221 28555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:47.041404 28555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:04:47.052476 28555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:33773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=46637
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:47.053728 28555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:47.055063 28555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:47.067577 28561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:47.067553 28560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:47.067585 28563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:47.068678 28555 server_base.cc:1034] running on GCE node
I20260924 10:04:47.069273 28555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:47.070176 28555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:47.071383 28555 hybrid_clock.cc:648] HybridClock initialized: now 1790244287071316 us; error 58 us; skew 500 ppm
I20260924 10:04:47.071684 28555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:47 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244287, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:47.074877 28555 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:04:47.076339 28555 webserver.cc:460] Webserver started at http://127.13.162.2:46637/ using document root <none> and password file <none>
I20260924 10:04:47.077023 28555 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:47.077147 28555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:47.082571 28555 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:04:47.085404 28570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:47.087008 28555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:04:47.087173 28555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "8224135850df46ecafbde4fbb31da610"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "eb2ac725946704987da85faac7b2ff06"
server_key_iv: "f7eaf65fd4fd3f7c62480bf906c8ec88"
server_key_version: "encryptionkey@0"
I20260924 10:04:47.088202 28555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:47.114378 28555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:47.118623 28555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:47.118876 28555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:47.119921 28555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:47.121249 28555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:47.121352 28555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:47.121455 28555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:47.121536 28555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:47.151875 28555 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:33773
I20260924 10:04:47.151937 28682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:33773 every 8 connection(s)
I20260924 10:04:47.153281 28555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:04:47.160826 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28555
I20260924 10:04:47.161393 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:41077
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:37281
--webserver_port=42265
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:04:47 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244287, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:47.174554 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.155676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:46835 (local address 127.13.162.62:37281)
0924 10:04:47.155882 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.155887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.157059 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 10:04:47.158923 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.158929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.159012 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.159096 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:47.160602 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.161340 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.163553 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.163988 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.164095 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.168404 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:47.168409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:47.168412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:47.168449 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:47.171148 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:47.171979 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:47.171986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:47.171989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:47.172149 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:47.172611 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:47.172616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:47.172619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:47.173030 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:47.173167 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.173966 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.174217 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:04:47.175468 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.156210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:46835)
0924 10:04:47.156900 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.156945 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.158664 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.159336 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.159345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.159841 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:47.160408 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.160431 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.161495 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.161500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.163392 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.163408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.164226 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.164233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.164515 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.165331 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:47.165365 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:47.168125 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:47.171312 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:47.171320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:47.171335 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:47.171803 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:47.172276 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:47.172279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:47.172282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:47.172497 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:47.173174 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:47.173181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:47.173566 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.173802 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.174281 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":248,"threads_started":1}
I20260924 10:04:47.176946 28683 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37281
I20260924 10:04:47.177277 28683 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:47.178018 28683 heartbeater.cc:511] Master 127.13.162.62:37281 requested a full tablet report, sending...
I20260924 10:04:47.179445 28366 ts_manager.cc:194] Registered new tserver with Master: 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773)
I20260924 10:04:47.180274 28366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:46835
W20260924 10:04:47.494141 28686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:47.494460 28686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:47.494560 28686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:47.502239 28686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:47.502400 28686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:47.502537 28686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:04:47.512388 28686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=42265
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:47.513654 28686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:47.515026 28686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:47.528258 28694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:47.528187 28691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:47.528271 28692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:47.529119 28686 server_base.cc:1034] running on GCE node
I20260924 10:04:47.529771 28686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:47.530856 28686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:47.532092 28686 hybrid_clock.cc:648] HybridClock initialized: now 1790244287532033 us; error 47 us; skew 500 ppm
I20260924 10:04:47.532444 28686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:47 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244287, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:47.535622 28686 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:04:47.537127 28686 webserver.cc:460] Webserver started at http://127.13.162.3:42265/ using document root <none> and password file <none>
I20260924 10:04:47.537779 28686 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:47.537904 28686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:47.543473 28686 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:04:47.546197 28701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:47.547694 28686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:04:47.547878 28686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "5affdd7815fc4668a2d73d056683f098"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "314a6006d50c3f0feb4d91ff7eb28634"
server_key_iv: "de475da33ceb51ffaf318ad97d471181"
server_key_version: "encryptionkey@0"
I20260924 10:04:47.548888 28686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:47.567833 28686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:47.572382 28686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:47.572655 28686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:47.573724 28686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:47.575146 28686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:47.575244 28686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:47.575350 28686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:47.575423 28686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:47.606726 28686 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:41077
I20260924 10:04:47.606789 28813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:41077 every 8 connection(s)
I20260924 10:04:47.608280 28686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:04:47.615785 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28686
Sep 24 10:04:47 dist-test-slave-jvsj krb5kdc[27782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244287, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:04:47.631268 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.610946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:44221 (local address 127.13.162.62:37281)
0924 10:04:47.611169 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.611175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.612447 (+ 1272us) server_negotiation.cc:407] Connection header received
0924 10:04:47.614272 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.614276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.614396 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.614497 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:47.616148 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.617264 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.619767 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.620565 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.620696 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.624868 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:47.624873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:47.624876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:47.624909 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:47.627783 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:47.628709 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:47.628713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:47.628715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:47.628843 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:47.629317 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:47.629322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:47.629325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:47.629695 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:47.629842 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.630696 (+ 854us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.630912 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":75}
I20260924 10:04:47.632625 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.611510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:44221)
0924 10:04:47.612268 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.612313 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.613973 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.614779 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.614788 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.615331 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:04:47.615969 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.615991 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.617479 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.617486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.619595 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.619614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.620784 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.620788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.620996 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.621726 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:04:47.621758 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:04:47.624621 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:04:47.627915 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:47.627925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:47.627945 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:47.628598 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:47.628945 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:04:47.628951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:04:47.628954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:04:47.629205 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:04:47.629822 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:04:47.629829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:04:47.630240 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.630601 (+ 361us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.631171 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260924 10:04:47.632922 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.616704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.13.162.62:37281)
0924 10:04:47.617200 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.617205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.617544 (+ 339us) server_negotiation.cc:407] Connection header received
0924 10:04:47.617754 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.617757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.617816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.617887 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:47.619821 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.620949 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.622824 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.623258 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.623414 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.626099 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:47.626104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:47.626107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:47.626154 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:47.629245 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:47.630101 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:47.630107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:47.630109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:47.630245 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:47.631478 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:47.631483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:47.631486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:47.631811 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:47.632075 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.632318 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.632537 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260924 10:04:47.634137 28814 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37281
I20260924 10:04:47.634490 28814 heartbeater.cc:464] Registering TS with master...
I20260924 10:04:47.635414 28814 heartbeater.cc:511] Master 127.13.162.62:37281 requested a full tablet report, sending...
I20260924 10:04:47.637043 28365 ts_manager.cc:194] Registered new tserver with Master: 5affdd7815fc4668a2d73d056683f098 (127.13.162.3:41077)
I20260924 10:04:47.637890 28365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:44221
I20260924 10:04:47.648474 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:04:47.655539 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.649392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.13.162.62:37281)
0924 10:04:47.649568 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.649572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.650128 (+ 556us) server_negotiation.cc:407] Connection header received
0924 10:04:47.650298 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.650302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.650504 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.650664 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:04:47.651047 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.651827 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.653512 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.653975 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.654073 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.654563 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:04:47.655098 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:04:47.655207 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.655276 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.655342 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:04:47.656524 28365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:04:47.663775 28365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:04:47.700721 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.675725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.13.162.2:33773)
0924 10:04:47.692648 (+ 16923us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.692657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.692699 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:04:47.692811 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.692819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.693314 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.693505 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:47.694297 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.695871 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.698259 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.699644 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.699762 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.699976 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.700255 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.700421 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16755,"thread_start_us":17169,"threads_started":1}
I20260924 10:04:47.700785 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.675418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:33773 (local address 127.0.0.1:46252)
0924 10:04:47.686682 (+ 11264us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.687922 (+ 1240us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.688197 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.693553 (+ 5356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.693563 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.693599 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:47.694083 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.694098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.695998 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.696003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.698103 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.698115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.699828 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.699835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.700022 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.700086 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.700202 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.700386 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10973,"mutex_wait_us":45,"thread_start_us":15637,"threads_started":1}
I20260924 10:04:47.702003 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.676093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38248)
0924 10:04:47.693085 (+ 16992us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.693103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.693217 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.694651 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.694655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.694681 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:47.695089 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.695101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.697109 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.697118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.699342 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.699356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.701191 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.701199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.701420 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.701452 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.701547 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.701667 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16857,"mutex_wait_us":65,"thread_start_us":16719,"threads_started":1}
I20260924 10:04:47.702724 28824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.675732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:41077 (local address 127.0.0.1:41952)
0924 10:04:47.691383 (+ 15651us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.691406 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.691550 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.694146 (+ 2596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.694151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.694175 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:47.694514 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.694525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.696478 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.696483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.698628 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.698643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.701357 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.701364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.701502 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.701532 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.702257 (+ 725us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.702406 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15510,"mutex_wait_us":6,"thread_start_us":10477,"threads_started":1}
I20260924 10:04:47.703605 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.676329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.13.162.3:41077)
0924 10:04:47.691620 (+ 15291us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.691632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.691671 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:04:47.691783 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.691791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.692106 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.693507 (+ 1401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:47.694786 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.696255 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.698805 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.700266 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.700386 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.700682 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.701696 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.703273 (+ 1577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15057,"thread_start_us":16113,"threads_started":1}
I20260924 10:04:47.704842 28829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.676606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.13.162.1:41995)
0924 10:04:47.694083 (+ 17477us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.694090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.694111 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:04:47.694206 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.694213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.694424 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.695283 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:47.695504 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.696971 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.699523 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.700980 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.701103 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.701266 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.701614 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.701784 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17348,"thread_start_us":17254,"threads_started":1}
I20260924 10:04:47.705169 28618 tablet_service.cc:1467] Processing CreateTablet for tablet dee8d8ece6e5449780950ad4f8145b8f (DEFAULT_TABLE table=test-table [id=be3d32234b8b4ab4a742016b5d57604c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:47.706661 28618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee8d8ece6e5449780950ad4f8145b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:47.707209 28749 tablet_service.cc:1467] Processing CreateTablet for tablet dee8d8ece6e5449780950ad4f8145b8f (DEFAULT_TABLE table=test-table [id=be3d32234b8b4ab4a742016b5d57604c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:47.708560 28749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee8d8ece6e5449780950ad4f8145b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:47.708670 28486 tablet_service.cc:1467] Processing CreateTablet for tablet dee8d8ece6e5449780950ad4f8145b8f (DEFAULT_TABLE table=test-table [id=be3d32234b8b4ab4a742016b5d57604c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:04:47.709988 28486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee8d8ece6e5449780950ad4f8145b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:04:47.719573 28830 tablet_bootstrap.cc:492] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Bootstrap starting.
I20260924 10:04:47.723414 28830 tablet_bootstrap.cc:654] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:47.724731 28830 log.cc:824] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:47.727821 28830 tablet_bootstrap.cc:492] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: No bootstrap required, opened a new log
I20260924 10:04:47.728119 28830 ts_tablet_manager.cc:1397] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 10:04:47.730554 28831 tablet_bootstrap.cc:492] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Bootstrap starting.
I20260924 10:04:47.734165 28832 tablet_bootstrap.cc:492] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Bootstrap starting.
I20260924 10:04:47.734711 28831 tablet_bootstrap.cc:654] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:47.735769 28831 log.cc:824] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:47.737272 28830 raft_consensus.cc:348] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.737771 28830 raft_consensus.cc:374] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:47.737874 28830 raft_consensus.cc:729] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8224135850df46ecafbde4fbb31da610, State: Initialized, Role: FOLLOWER
I20260924 10:04:47.738196 28832 tablet_bootstrap.cc:654] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Neither blocks nor log segments found. Creating new log.
I20260924 10:04:47.738600 28831 tablet_bootstrap.cc:492] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: No bootstrap required, opened a new log
I20260924 10:04:47.738575 28830 consensus_queue.cc:260] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.738844 28831 ts_tablet_manager.cc:1397] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 10:04:47.739478 28832 log.cc:824] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:47.739725 28683 heartbeater.cc:503] Master 127.13.162.62:37281 was elected leader, sending a full tablet report...
I20260924 10:04:47.740427 28830 ts_tablet_manager.cc:1428] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260924 10:04:47.741197 28551 heartbeater.cc:503] Master 127.13.162.62:37281 was elected leader, sending a full tablet report...
I20260924 10:04:47.743992 28832 tablet_bootstrap.cc:492] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: No bootstrap required, opened a new log
I20260924 10:04:47.744285 28832 ts_tablet_manager.cc:1397] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 10:04:47.748247 28831 raft_consensus.cc:348] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.748603 28831 raft_consensus.cc:374] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:47.748704 28831 raft_consensus.cc:729] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5affdd7815fc4668a2d73d056683f098, State: Initialized, Role: FOLLOWER
I20260924 10:04:47.749284 28831 consensus_queue.cc:260] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.750423 28814 heartbeater.cc:503] Master 127.13.162.62:37281 was elected leader, sending a full tablet report...
I20260924 10:04:47.750968 28831 ts_tablet_manager.cc:1428] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 10:04:47.752827 28832 raft_consensus.cc:348] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.753275 28832 raft_consensus.cc:374] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:04:47.753372 28832 raft_consensus.cc:729] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e668593e7d34ba4b8423cfd4c87df42, State: Initialized, Role: FOLLOWER
I20260924 10:04:47.754019 28832 consensus_queue.cc:260] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.755813 28832 ts_tablet_manager.cc:1428] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20260924 10:04:47.859993 28815 tablet.cc:2369] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:04:47.905181 28684 tablet.cc:2369] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:04:47.959630 28552 tablet.cc:2369] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:04:47.969314 28837 raft_consensus.cc:482] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:04:47.969606 28837 raft_consensus.cc:504] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.971376 28837 leader_election.cc:288] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773), 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1:41995)
I20260924 10:04:47.977747 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.971673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.13.162.3:53935)
0924 10:04:47.971904 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.971925 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.972079 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.972423 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.972429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.972465 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:47.972682 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.972693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.973881 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.973886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.975802 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.975817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.977151 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.977155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.977284 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.977309 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.977380 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.977475 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260924 10:04:47.977809 28829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.971838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:53935 (local address 127.13.162.1:41995)
0924 10:04:47.972092 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.972097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.972120 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:04:47.972196 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.972201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.972278 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.972371 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:47.972831 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.973768 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.975967 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.976992 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.977161 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.977238 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.977419 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.977557 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 10:04:47.978519 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.971976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46333 (local address 127.13.162.2:33773)
0924 10:04:47.972190 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:47.972196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:47.972761 (+ 565us) server_negotiation.cc:407] Connection header received
0924 10:04:47.972895 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:47.972900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:47.972957 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:47.973048 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:04:47.973569 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.974285 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.976387 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.977530 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.977678 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.977746 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:47.978135 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 10:04:47.978262 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:04:47.978543 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:47.971743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:33773 (local address 127.13.162.3:46333)
0924 10:04:47.972637 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:47.972651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:04:47.972771 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:04:47.973090 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:04:47.973095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:04:47.973120 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:04:47.973414 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.973424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.974458 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.974464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.976239 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:04:47.976250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:04:47.977739 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:04:47.977745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:04:47.977909 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:47.977942 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:04:47.978084 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:04:47.978208 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":575,"threads_started":1}
I20260924 10:04:47.978762 28506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8d8ece6e5449780950ad4f8145b8f" candidate_uuid: "5affdd7815fc4668a2d73d056683f098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e668593e7d34ba4b8423cfd4c87df42" is_pre_election: true
I20260924 10:04:47.979315 28506 raft_consensus.cc:2454] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5affdd7815fc4668a2d73d056683f098 in term 0.
I20260924 10:04:47.979374 28638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8d8ece6e5449780950ad4f8145b8f" candidate_uuid: "5affdd7815fc4668a2d73d056683f098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8224135850df46ecafbde4fbb31da610" is_pre_election: true
I20260924 10:04:47.979863 28638 raft_consensus.cc:2454] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5affdd7815fc4668a2d73d056683f098 in term 0.
I20260924 10:04:47.979967 28704 leader_election.cc:302] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5affdd7815fc4668a2d73d056683f098, 9e668593e7d34ba4b8423cfd4c87df42; no voters:
I20260924 10:04:47.980350 28837 raft_consensus.cc:2781] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:04:47.980469 28837 raft_consensus.cc:482] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:04:47.980576 28837 raft_consensus.cc:3037] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:47.982707 28837 raft_consensus.cc:504] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.983273 28837 leader_election.cc:288] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [CANDIDATE]: Term 1 election: Requested vote from peers 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773), 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1:41995)
I20260924 10:04:47.983721 28638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8d8ece6e5449780950ad4f8145b8f" candidate_uuid: "5affdd7815fc4668a2d73d056683f098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8224135850df46ecafbde4fbb31da610"
I20260924 10:04:47.983742 28506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee8d8ece6e5449780950ad4f8145b8f" candidate_uuid: "5affdd7815fc4668a2d73d056683f098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e668593e7d34ba4b8423cfd4c87df42"
I20260924 10:04:47.983922 28638 raft_consensus.cc:3037] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:47.983922 28506 raft_consensus.cc:3037] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:04:47.986990 28506 raft_consensus.cc:2454] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5affdd7815fc4668a2d73d056683f098 in term 1.
I20260924 10:04:47.986990 28638 raft_consensus.cc:2454] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5affdd7815fc4668a2d73d056683f098 in term 1.
I20260924 10:04:47.987418 28705 leader_election.cc:302] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5affdd7815fc4668a2d73d056683f098, 8224135850df46ecafbde4fbb31da610; no voters:
I20260924 10:04:47.987758 28837 raft_consensus.cc:2781] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:04:47.988240 28837 raft_consensus.cc:686] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 1 LEADER]: Becoming Leader. State: Replica: 5affdd7815fc4668a2d73d056683f098, State: Running, Role: LEADER
I20260924 10:04:47.988807 28837 consensus_queue.cc:237] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } }
I20260924 10:04:47.993499 28366 catalog_manager.cc:5495] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5affdd7815fc4668a2d73d056683f098 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "5affdd7815fc4668a2d73d056683f098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5affdd7815fc4668a2d73d056683f098" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 41077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 } health_report { overall_health: UNKNOWN } } }
I20260924 10:04:48.071586 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.066479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.13.162.3:41077)
0924 10:04:48.066632 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:48.066636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:48.066708 (+ 72us) server_negotiation.cc:407] Connection header received
0924 10:04:48.066876 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:48.066879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:48.066955 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:48.067079 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:04:48.067450 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.068160 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.069495 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.069894 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.069998 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:48.070530 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:04:48.071045 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:04:48.071161 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:48.071304 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 10:04:48.071379 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:04:48.079797 28506 raft_consensus.cc:1261] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Refusing update from remote peer 5affdd7815fc4668a2d73d056683f098: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:48.079797 28638 raft_consensus.cc:1261] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Refusing update from remote peer 5affdd7815fc4668a2d73d056683f098: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:04:48.080818 28840 consensus_queue.cc:1035] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e668593e7d34ba4b8423cfd4c87df42" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:48.081175 28837 consensus_queue.cc:1035] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8224135850df46ecafbde4fbb31da610" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 33773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:04:48.092702 28843 mvcc.cc:205] Tried to move back new op lower bound from 7332840603964280832 to 7332840603605909504. Current Snapshot: MvccSnapshot[applied={T|T < 7332840603964280832}]
I20260924 10:04:48.106895 28365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:04:48.107214 28365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:04:48.110913 28365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Sending DeleteTablet for 3 replicas of tablet dee8d8ece6e5449780950ad4f8145b8f
I20260924 10:04:48.112102 28749 tablet_service.cc:1514] Processing DeleteTablet for tablet dee8d8ece6e5449780950ad4f8145b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:48 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:41952
I20260924 10:04:48.112124 28486 tablet_service.cc:1514] Processing DeleteTablet for tablet dee8d8ece6e5449780950ad4f8145b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:48 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:38248
I20260924 10:04:48.112865 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28403
I20260924 10:04:48.112834 28618 tablet_service.cc:1514] Processing DeleteTablet for tablet dee8d8ece6e5449780950ad4f8145b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:04:48 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:46252
I20260924 10:04:48.122202 28854 tablet_replica.cc:330] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: stopping tablet replica
I20260924 10:04:48.122802 28854 raft_consensus.cc:2229] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:04:48.123447 28854 raft_consensus.cc:2258] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:48.125253 28855 tablet_replica.cc:330] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: stopping tablet replica
I20260924 10:04:48.126080 28855 raft_consensus.cc:2229] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:04:48.127099 28855 raft_consensus.cc:2258] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:48.127228 28853 tablet_replica.cc:330] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: stopping tablet replica
I20260924 10:04:48.127676 28853 raft_consensus.cc:2229] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:04:48.127949 28854 ts_tablet_manager.cc:1905] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:48.131001 28853 raft_consensus.cc:2258] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:04:48.131963 28855 ts_tablet_manager.cc:1905] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:48.132718 28829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.114459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.13.162.1:41995)
0924 10:04:48.114637 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:48.114642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:48.114660 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:04:48.114714 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:48.114720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:48.114803 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:48.114899 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:48.117038 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.118196 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.120167 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.120832 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.120967 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:48.124681 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:48.124694 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:48.124714 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:48.124758 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:48.128742 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:48.129568 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:48.129578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:48.129581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:48.129783 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:48.130210 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:48.130215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:48.130218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:48.130755 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:48.131058 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:48.131366 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:04:48.131601 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:04:48.133682 28546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:48.135561 28853 ts_tablet_manager.cc:1905] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:48.140959 28855 ts_tablet_manager.cc:1918] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:48.141152 28855 log.cc:1196] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/dee8d8ece6e5449780950ad4f8145b8f
I20260924 10:04:48.141752 28855 ts_tablet_manager.cc:1939] T dee8d8ece6e5449780950ad4f8145b8f P 8224135850df46ecafbde4fbb31da610: Deleting consensus metadata
I20260924 10:04:48.142899 28853 ts_tablet_manager.cc:1918] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:48.143071 28853 log.cc:1196] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/dee8d8ece6e5449780950ad4f8145b8f
I20260924 10:04:48.143627 28853 ts_tablet_manager.cc:1939] T dee8d8ece6e5449780950ad4f8145b8f P 5affdd7815fc4668a2d73d056683f098: Deleting consensus metadata
I20260924 10:04:48.143898 28353 catalog_manager.cc:4841] TS 8224135850df46ecafbde4fbb31da610 (127.13.162.2:33773): tablet dee8d8ece6e5449780950ad4f8145b8f (table test-table [id=be3d32234b8b4ab4a742016b5d57604c]) successfully deleted
I20260924 10:04:48.145346 28350 catalog_manager.cc:4841] TS 5affdd7815fc4668a2d73d056683f098 (127.13.162.3:41077): tablet dee8d8ece6e5449780950ad4f8145b8f (table test-table [id=be3d32234b8b4ab4a742016b5d57604c]) successfully deleted
I20260924 10:04:48.261945 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28403
I20260924 10:04:48.277171 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28555
W20260924 10:04:48.277212 28352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:41995 (error 108)
I20260924 10:04:48.282437 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.279505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38262)
0924 10:04:48.282011 (+ 2506us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.282157 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2371,"thread_start_us":2278,"threads_started":1}
W20260924 10:04:48.282763 28352 catalog_manager.cc:4584] TS 9e668593e7d34ba4b8423cfd4c87df42 (127.13.162.1:41995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dee8d8ece6e5449780950ad4f8145b8f: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
I20260924 10:04:48.293741 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.278050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.13.162.2:33773)
0924 10:04:48.278241 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:48.278247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:48.278379 (+ 132us) server_negotiation.cc:407] Connection header received
0924 10:04:48.278580 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:48.278586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:48.278669 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:48.278773 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:48.280474 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.281562 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.283305 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.283989 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.284127 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:48.286853 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:48.286876 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:48.286899 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:48.286947 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:48.290716 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:48.291479 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:48.291489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:48.291493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:48.291666 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:48.292012 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:48.292017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:48.292020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:48.292541 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:48.292846 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:48.293094 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 10:04:48.293323 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:04:48.294494 28678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:48.336652 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.336101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38268)
0924 10:04:48.336352 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.336460 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 10:04:48.337877 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.337434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38270)
0924 10:04:48.337620 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.337719 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:04:48.383004 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28555
I20260924 10:04:48.398141 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28686
I20260924 10:04:48.398473 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.398002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38282)
0924 10:04:48.398194 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.398285 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:04:48.399726 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.399337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38290)
0924 10:04:48.399469 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.399548 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 10:04:48.414914 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.399056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.13.162.3:41077)
0924 10:04:48.399243 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:04:48.399249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:04:48.399313 (+ 64us) server_negotiation.cc:407] Connection header received
0924 10:04:48.399532 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:04:48.399538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:04:48.399650 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:04:48.399766 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:04:48.401562 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.402467 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.404321 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:04:48.404780 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:04:48.404876 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:04:48.408370 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:04:48.408385 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:04:48.408405 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:04:48.408453 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:04:48.411669 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:48.412644 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:48.412652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:48.412654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:48.412830 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:04:48.413271 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:04:48.413274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:04:48.413277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:04:48.413661 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:04:48.413946 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 10:04:48.414286 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 10:04:48.414537 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:04:48.415699 28809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:48.480568 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.480033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38300)
0924 10:04:48.480286 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.480386 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 10:04:48.481709 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.481266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38316)
0924 10:04:48.481486 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.481570 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260924 10:04:48.541138 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28686
I20260924 10:04:48.557327 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28333
I20260924 10:04:48.558578 28396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:04:48.672515 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:48.672014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:41995 (local address 127.0.0.1:38320)
0924 10:04:48.672207 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:48.672330 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:04:48.672470 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28333
I20260924 10:04:48.691386 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37281
--webserver_interface=127.13.162.62
--webserver_port=41341
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:49.022141 28862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:49.022496 28862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:49.022606 28862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:49.029811 28862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:04:49.029927 28862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:49.029971 28862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:49.030027 28862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:04:49.030071 28862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:04:49.040313 28862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=41341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:49.041633 28862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:49.042999 28862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:49.054785 28867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:49.054804 28870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:49.054896 28868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:49.055878 28862 server_base.cc:1034] running on GCE node
I20260924 10:04:49.056639 28862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:49.058034 28862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:49.059240 28862 hybrid_clock.cc:648] HybridClock initialized: now 1790244289059182 us; error 43 us; skew 500 ppm
I20260924 10:04:49.059595 28862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:49 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244289, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:49.062624 28862 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:04:49.064080 28862 webserver.cc:460] Webserver started at http://127.13.162.62:41341/ using document root <none> and password file <none>
I20260924 10:04:49.064720 28862 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:49.064834 28862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:49.070102 28862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:04:49.072976 28877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:49.074309 28862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:04:49.074499 28862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "76341fcaaa1a4866b4b1bd9ea5209006"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "f2dc4206a1fc08e0866b1e5887ec0258"
server_key_iv: "9eb31068ef475a6f4d59df3331506087"
server_key_version: "encryptionkey@0"
I20260924 10:04:49.075407 28862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:49.092717 28862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:49.096662 28862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:49.096884 28862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:49.113988 28862 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:37281
I20260924 10:04:49.114009 28928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:37281 every 8 connection(s)
I20260924 10:04:49.115513 28862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:04:49.122545 28929 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:04:49.122838 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28862
I20260924 10:04:49.123781 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:41995
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:37281
--webserver_port=40569
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:04:49.127398 28929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap starting.
I20260924 10:04:49.148893 28929 log.cc:824] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Log is configured to *not* fsync() on all Append() calls
I20260924 10:04:49.165556 28929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:04:49.166122 28929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006: Bootstrap complete.
I20260924 10:04:49.175652 28929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:49.176581 28929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76341fcaaa1a4866b4b1bd9ea5209006, State: Initialized, Role: FOLLOWER
I20260924 10:04:49.177289 28929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:49.177513 28929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:04:49.177619 28929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:04:49.177770 28929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:04:49.181910 28929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:49.182427 28929 leader_election.cc:302] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76341fcaaa1a4866b4b1bd9ea5209006; no voters:
I20260924 10:04:49.183068 28929 leader_election.cc:288] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:04:49.183211 28933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:04:49.184247 28933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [term 3 LEADER]: Becoming Leader. State: Replica: 76341fcaaa1a4866b4b1bd9ea5209006, State: Running, Role: LEADER
I20260924 10:04:49.184829 28933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } }
I20260924 10:04:49.185235 28929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:04:49.187496 28934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } } }
I20260924 10:04:49.187487 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76341fcaaa1a4866b4b1bd9ea5209006. Latest consensus state: current_term: 3 leader_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76341fcaaa1a4866b4b1bd9ea5209006" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37281 } } }
I20260924 10:04:49.187908 28934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:49.187912 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76341fcaaa1a4866b4b1bd9ea5209006 [sys.catalog]: This master's current role is: LEADER
I20260924 10:04:49.188853 28938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:04:49.208978 28938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:04:49.210814 28938 catalog_manager.cc:1259] Loaded cluster ID: bf0c14dbbf6848f884e605b500904380
I20260924 10:04:49.210976 28938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 10:04:49.213270 28938 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f23e3be49ad google::LogMessage::Fail() at ??:0
@ 0x7f23e3be788a google::LogMessage::SendToLog() at ??:0
@ 0x7f23e3be44de google::LogMessage::Flush() at ??:0
@ 0x7f23e3be5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f23f961b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f23f9618976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f23f96a0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f23f96a0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f23e6830345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f23e683391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f23e6833799 std::_Function_handler<>::_M_invoke() at ??:0
I20260924 10:04:49.270850 28958 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f23e680eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f23e9ae86db start_thread at ??:0
@ 0x7f23e2a7e61f clone at ??:0
W20260924 10:04:49.508397 28931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:49.508688 28931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:49.508813 28931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:49.516390 28931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:49.516500 28931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:49.516649 28931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:04:49.526651 28931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:41995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=40569
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:49.528080 28931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:49.529513 28931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:49.542632 28970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:49.542555 28971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:49.542590 28973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:49.543422 28931 server_base.cc:1034] running on GCE node
I20260924 10:04:49.544093 28931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:49.545092 28931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:49.546307 28931 hybrid_clock.cc:648] HybridClock initialized: now 1790244289546248 us; error 46 us; skew 500 ppm
I20260924 10:04:49.546646 28931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:49 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244289, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:49.549759 28931 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:04:49.551350 28931 webserver.cc:460] Webserver started at http://127.13.162.1:40569/ using document root <none> and password file <none>
I20260924 10:04:49.552052 28931 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:49.552174 28931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:49.557850 28931 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:04:49.560969 28980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:49.562542 28931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:04:49.562700 28931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "9e668593e7d34ba4b8423cfd4c87df42"
format_stamp: "Formatted at 2026-09-24 10:04:43 on dist-test-slave-jvsj"
server_key: "caf8e9b621afefae89d9e305f9bdb3f0"
server_key_iv: "5f0f6a5f558e4728f9e10ae26a2852d0"
server_key_version: "encryptionkey@0"
I20260924 10:04:49.563671 28931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:49.583657 28931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:49.587601 28931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:49.587842 28931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:49.588806 28931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:49.590751 28987 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 10:04:49.596096 28987 ts_tablet_manager.cc:1852] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 10:04:49.597114 28987 ts_tablet_manager.cc:1905] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:04:49.605089 28987 ts_tablet_manager.cc:1918] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:04:49.605312 28987 log.cc:1196] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/dee8d8ece6e5449780950ad4f8145b8f
I20260924 10:04:49.605772 28987 ts_tablet_manager.cc:1939] T dee8d8ece6e5449780950ad4f8145b8f P 9e668593e7d34ba4b8423cfd4c87df42: Deleting consensus metadata
I20260924 10:04:49.606178 28931 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 10:04:49.606367 28931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260924 10:04:49.606498 28931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:49.606576 28931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:49.637724 28931 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:41995
I20260924 10:04:49.637777 29093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:41995 every 8 connection(s)
I20260924 10:04:49.639070 28931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:04:49.643949 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:49.642047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:39855)
0924 10:04:49.642822 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:49.643083 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
W20260924 10:04:49.644558 29094 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:37281 (0 consecutive failures): Network error: Failed to ping master at 127.13.162.62:37281: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
I20260924 10:04:49.645421 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:49.645085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:42783)
0924 10:04:49.645239 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:49.645313 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:04:49.646253 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:49.645923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:43151)
0924 10:04:49.646059 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:49.646133 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:04:49.646507 29094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:04:49.649125 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 28931
I20260924 10:04:49.649693 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:33773
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:37281
--webserver_port=46637
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:49.974613 29097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:49.974903 29097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:49.975013 29097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:49.982234 29097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:49.982372 29097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:49.982507 29097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:04:49.992265 29097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:33773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=46637
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:49.993564 29097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:49.994940 29097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:50.008363 29105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:50.008406 29102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:50.008433 29103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:50.009264 29097 server_base.cc:1034] running on GCE node
I20260924 10:04:50.009986 29097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:50.011057 29097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:50.012291 29097 hybrid_clock.cc:648] HybridClock initialized: now 1790244290012231 us; error 49 us; skew 500 ppm
I20260924 10:04:50.012611 29097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:50 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244290, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:50.021759 29097 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:04:50.023414 29097 webserver.cc:460] Webserver started at http://127.13.162.2:46637/ using document root <none> and password file <none>
I20260924 10:04:50.024122 29097 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:50.024235 29097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:50.029711 29097 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 10:04:50.032814 29112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:50.034253 29097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:04:50.034433 29097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "8224135850df46ecafbde4fbb31da610"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "eb2ac725946704987da85faac7b2ff06"
server_key_iv: "f7eaf65fd4fd3f7c62480bf906c8ec88"
server_key_version: "encryptionkey@0"
I20260924 10:04:50.035415 29097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:50.066083 29097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:50.070526 29097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:50.070762 29097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:50.071727 29097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:50.073101 29097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:50.073185 29097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:50.073287 29097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:50.073351 29097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:50.103489 29097 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:33773
I20260924 10:04:50.103559 29224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:33773 every 8 connection(s)
I20260924 10:04:50.104871 29097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:04:50.109475 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.107730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:43439)
0924 10:04:50.108418 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.108640 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
W20260924 10:04:50.110106 29225 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:37281 (0 consecutive failures): Network error: Failed to ping master at 127.13.162.62:37281: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
I20260924 10:04:50.110894 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.110598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:48491)
0924 10:04:50.110730 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.110796 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:04:50.111509 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29097
I20260924 10:04:50.111670 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.111370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:33157)
0924 10:04:50.111485 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.111551 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:04:50.111907 29225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:04:50.112177 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:41077
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:37281
--webserver_port=42265
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:04:50.436794 29228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:04:50.437114 29228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:04:50.437254 29228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:04:50.444821 29228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:04:50.444964 29228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:04:50.445128 29228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:04:50.455457 29228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=42265
--tserver_master_addrs=127.13.162.62:37281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:04:50.456789 29228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:04:50.458099 29228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:04:50.472043 29233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:50.472087 29234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:04:50.472008 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:04:50.473452 29228 server_base.cc:1034] running on GCE node
I20260924 10:04:50.474149 29228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:04:50.475200 29228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:04:50.476426 29228 hybrid_clock.cc:648] HybridClock initialized: now 1790244290476377 us; error 40 us; skew 500 ppm
I20260924 10:04:50.476780 29228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:04:50 dist-test-slave-jvsj krb5kdc[27782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244290, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:04:50.479962 29228 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:04:50.481442 29228 webserver.cc:460] Webserver started at http://127.13.162.3:42265/ using document root <none> and password file <none>
I20260924 10:04:50.482153 29228 fs_manager.cc:359] Metadata directory not provided
I20260924 10:04:50.482275 29228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:04:50.487951 29228 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:04:50.490998 29243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:50.492381 29228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:04:50.492569 29228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "5affdd7815fc4668a2d73d056683f098"
format_stamp: "Formatted at 2026-09-24 10:04:44 on dist-test-slave-jvsj"
server_key: "314a6006d50c3f0feb4d91ff7eb28634"
server_key_iv: "de475da33ceb51ffaf318ad97d471181"
server_key_version: "encryptionkey@0"
I20260924 10:04:50.493563 29228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:04:50.513813 29228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:04:50.517902 29228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:04:50.518185 29228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:04:50.519234 29228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:04:50.520586 29228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:04:50.520681 29228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:50.520793 29228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:04:50.520872 29228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:04:50.552350 29228 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:41077
I20260924 10:04:50.552400 29355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:41077 every 8 connection(s)
I20260924 10:04:50.553876 29228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:04:50.555734 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29228
I20260924 10:04:50.559101 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.557093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:34251)
0924 10:04:50.557919 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.558179 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":340,"threads_started":1}
W20260924 10:04:50.559778 29356 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:37281 (0 consecutive failures): Network error: Failed to ping master at 127.13.162.62:37281: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
I20260924 10:04:50.560559 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.560250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:43785)
0924 10:04:50.560384 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.560451 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:04:50.561301 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.561015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:53927)
0924 10:04:50.561126 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.561198 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:04:50.561532 29356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:04:50.648015 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:50.647160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:47797)
0924 10:04:50.647642 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:50.647768 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260924 10:04:51.113480 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:51.112593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:48313)
0924 10:04:51.113124 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:51.113239 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":237,"threads_started":1}
I20260924 10:04:51.563023 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:51.562125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:60721)
0924 10:04:51.562679 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:51.562794 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":272,"threads_started":1}
I20260924 10:04:51.649978 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:51.649178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:50021)
0924 10:04:51.649725 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:51.649822 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":281,"threads_started":1}
I20260924 10:04:52.115319 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:52.114526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:47707)
0924 10:04:52.115066 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:52.115147 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":261,"threads_started":1}
I20260924 10:04:52.564718 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:52.563979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:55403)
0924 10:04:52.564450 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:52.564541 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260924 10:04:52.651844 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:52.651032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:34849)
0924 10:04:52.651539 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:52.651659 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":258,"threads_started":1}
I20260924 10:04:53.117017 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:53.116206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:59053)
0924 10:04:53.116721 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:53.116812 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260924 10:04:53.566725 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:53.565840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:56767)
0924 10:04:53.566438 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:53.566534 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":317,"threads_started":1}
I20260924 10:04:53.653596 29369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:53.652830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:55331)
0924 10:04:53.653335 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:53.653425 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":249,"threads_started":1}
I20260924 10:04:54.118957 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:54.118123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:33987)
0924 10:04:54.118687 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:54.118782 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
I20260924 10:04:54.568485 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:54.567735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:51833)
0924 10:04:54.568185 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:54.568280 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":204,"threads_started":1}
I20260924 10:04:54.655405 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:54.654627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:44561)
0924 10:04:54.655107 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:54.655201 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":235,"threads_started":1}
I20260924 10:04:55.120747 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:55.119932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:45433)
0924 10:04:55.120476 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:55.120573 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":303,"threads_started":1}
I20260924 10:04:55.570399 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:55.569595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:59983)
0924 10:04:55.570086 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:55.570177 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":247,"threads_started":1}
I20260924 10:04:55.657151 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:55.656444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:43725)
0924 10:04:55.656895 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:55.656988 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":219,"threads_started":1}
I20260924 10:04:56.122401 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:56.121665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:57113)
0924 10:04:56.122101 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:56.122191 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260924 10:04:56.572429 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:56.571448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:49387)
0924 10:04:56.572008 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:56.572131 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":266,"threads_started":1}
I20260924 10:04:56.659189 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:56.658308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:35293)
0924 10:04:56.658888 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:56.658980 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":233,"threads_started":1}
I20260924 10:04:57.124377 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:57.123416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:49683)
0924 10:04:57.123932 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:57.124058 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":218,"threads_started":1}
I20260924 10:04:57.574370 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:57.573368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:37859)
0924 10:04:57.573954 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:57.574049 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":286,"threads_started":1}
I20260924 10:04:57.660866 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:57.660142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:49911)
0924 10:04:57.660605 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:57.660692 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260924 10:04:58.126196 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:58.125443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:58339)
0924 10:04:58.125933 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:58.126029 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260924 10:04:58.576150 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:58.575400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:49581)
0924 10:04:58.575879 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:58.575982 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":219,"threads_started":1}
I20260924 10:04:58.662593 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:58.661801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:50683)
0924 10:04:58.662270 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:58.662423 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260924 10:04:59.128105 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:59.127182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:36815)
0924 10:04:59.127786 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:59.127929 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":267,"threads_started":1}
I20260924 10:04:59.577850 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:59.577093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:53313)
0924 10:04:59.577558 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:59.577652 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260924 10:04:59.664265 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:04:59.663552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:57125)
0924 10:04:59.664010 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:04:59.664095 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":230,"threads_started":1}
I20260924 10:05:00.129911 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:00.129138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:58999)
0924 10:05:00.129650 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:00.129744 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":298,"threads_started":1}
I20260924 10:05:00.579597 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:00.578867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:48959)
0924 10:05:00.579332 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:00.579429 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":254,"threads_started":1}
I20260924 10:05:00.666137 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:00.665301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:60609)
0924 10:05:00.665848 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:00.665947 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":301,"threads_started":1}
I20260924 10:05:01.131932 29391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:01.131023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:45863)
0924 10:05:01.131594 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:01.131708 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":292,"threads_started":1}
I20260924 10:05:01.581403 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:01.580627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:51203)
0924 10:05:01.581110 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:01.581207 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":250,"threads_started":1}
I20260924 10:05:01.667912 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:01.667144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:38949)
0924 10:05:01.667643 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:01.667730 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":282,"threads_started":1}
I20260924 10:05:02.133875 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:02.133069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:51943)
0924 10:05:02.133589 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:02.133692 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":260,"threads_started":1}
I20260924 10:05:02.583287 29395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:02.582549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:35821)
0924 10:05:02.583035 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:02.583125 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260924 10:05:02.669688 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:02.668846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:53111)
0924 10:05:02.669384 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:02.669499 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
I20260924 10:05:03.135864 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:03.135001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:44061)
0924 10:05:03.135556 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:03.135663 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":242,"threads_started":1}
I20260924 10:05:03.585155 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:03.584320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:47885)
0924 10:05:03.584857 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:03.584955 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":273,"threads_started":1}
I20260924 10:05:03.671437 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:03.670596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:34123)
0924 10:05:03.671139 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:03.671224 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":302,"threads_started":1}
I20260924 10:05:04.137702 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:04.136933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:58299)
0924 10:05:04.137428 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:04.137522 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260924 10:05:04.586941 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:04.586146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:52905)
0924 10:05:04.586657 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:04.586741 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260924 10:05:04.673259 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:04.672514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:58573)
0924 10:05:04.672985 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:04.673077 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
I20260924 10:05:05.139858 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:05.139073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.2:34353)
0924 10:05:05.139574 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:05.139672 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260924 10:05:05.568148 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 28931
I20260924 10:05:05.588827 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:05.588122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.3:34885)
0924 10:05:05.588555 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:05.588647 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":208,"threads_started":1}
I20260924 10:05:05.592428 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:05.569400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.13.162.1:41995)
0924 10:05:05.570005 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:05.570028 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:05.570078 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:05:05.571863 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:05.571874 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:05.572328 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:05.572629 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:05.574010 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:05.575313 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:05.576605 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:05.577309 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:05.577451 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:05.585652 (+ 8201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:05.585661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:05.585677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:05.585723 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:05.589191 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:05.589905 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:05.589915 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:05.589925 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:05.590155 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:05.590575 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:05.590579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:05.590581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:05.591150 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:05.591437 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:05.591753 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:05:05.592103 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":319,"threads_started":1}
I20260924 10:05:05.593793 29089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:05.679890 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:05.678809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37281 (local address 127.13.162.1:38153)
0924 10:05:05.679362 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:05.679585 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":248,"threads_started":1}
I20260924 10:05:05.680258 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 28931
I20260924 10:05:05.695070 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29097
I20260924 10:05:05.712883 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:05.696126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.13.162.2:33773)
0924 10:05:05.696707 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:05.696721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:05.696760 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:05:05.698492 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:05.698503 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:05.698960 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:05.699337 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:05.700864 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:05.702151 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:05.703398 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:05.704073 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:05.704241 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:05.706883 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:05.706898 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:05.706914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:05.706954 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:05.709662 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:05.710332 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:05.710341 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:05.710350 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:05.710529 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:05.710885 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:05.710897 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:05.710899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:05.711433 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:05.711749 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:05.712180 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 10:05:05.712591 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":275,"threads_started":1}
I20260924 10:05:05.714289 29220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:05.797817 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29097
I20260924 10:05:05.812652 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29228
I20260924 10:05:05.830415 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:05.813701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.13.162.3:41077)
0924 10:05:05.814294 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:05.814308 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:05.814392 (+ 84us) server_negotiation.cc:407] Connection header received
0924 10:05:05.816066 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:05.816078 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:05.816542 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:05.816911 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:05.818415 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:05.819671 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:05.820953 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:05.821643 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:05.821827 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:05.824409 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:05.824417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:05.824433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:05.824471 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:05.827134 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:05.827824 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:05.827831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:05.827837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:05.828010 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:05.828383 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:05.828386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:05.828389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:05.828916 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:05.829231 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:05.829666 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 10:05:05.830086 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":262,"threads_started":1}
I20260924 10:05:05.831750 29351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:05.915189 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29228
2026-09-24T10:05:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:05:06 dist-test-slave-jvsj krb5kdc[29414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:05:06 dist-test-slave-jvsj krb5kdc[29414](info): set up 2 sockets
Sep 24 10:05:06 dist-test-slave-jvsj krb5kdc[29414](info): commencing operation
krb5kdc: starting...
W20260924 10:05:06.922809 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:05:07 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:05:07.149348 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45263
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:05:07.476927 29427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:07.477236 29427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:07.477324 29427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:07.484617 29427 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:05:07.484719 29427 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:05:07.484767 29427 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:05:07.484822 29427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:05:07.484879 29427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:07.484949 29427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:07.484997 29427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:05:07.485054 29427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:05:07.495056 29427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:07.496328 29427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:07.497717 29427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:07.509850 29432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:07.509876 29433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:07.509917 29435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:07.510675 29427 server_base.cc:1034] running on GCE node
I20260924 10:05:07.511477 29427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:07.512910 29427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:07.514138 29427 hybrid_clock.cc:648] HybridClock initialized: now 1790244307514075 us; error 43 us; skew 500 ppm
I20260924 10:05:07.514561 29427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:07 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244307, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:07.517616 29427 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:05:07.519135 29427 webserver.cc:460] Webserver started at http://127.13.162.62:40591/ using document root <none> and password file <none>
I20260924 10:05:07.519833 29427 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:07.519932 29427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:07.520238 29427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:05:07.523344 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "df0f317e5544467bb59d733201ef0881"
format_stamp: "Formatted at 2026-09-24 10:05:07 on dist-test-slave-jvsj"
server_key: "396ca37e66ba6c54169cc2ed0a221c88"
server_key_iv: "59ffd1422a802d8c511b3f5bc5746b58"
server_key_version: "encryptionkey@0"
I20260924 10:05:07.524147 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "df0f317e5544467bb59d733201ef0881"
format_stamp: "Formatted at 2026-09-24 10:05:07 on dist-test-slave-jvsj"
server_key: "396ca37e66ba6c54169cc2ed0a221c88"
server_key_iv: "59ffd1422a802d8c511b3f5bc5746b58"
server_key_version: "encryptionkey@0"
I20260924 10:05:07.529536 29427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:05:07.533329 29442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:07.534904 29427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:07.535082 29427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "df0f317e5544467bb59d733201ef0881"
format_stamp: "Formatted at 2026-09-24 10:05:07 on dist-test-slave-jvsj"
server_key: "396ca37e66ba6c54169cc2ed0a221c88"
server_key_iv: "59ffd1422a802d8c511b3f5bc5746b58"
server_key_version: "encryptionkey@0"
I20260924 10:05:07.535295 29427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:07.555261 29427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:07.559362 29427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:07.559614 29427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:07.577414 29427 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:45263
I20260924 10:05:07.577430 29493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:45263 every 8 connection(s)
I20260924 10:05:07.579030 29427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:05:07.580720 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29427
I20260924 10:05:07.580942 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:05:07.581794 13960 external_mini_cluster.cc:1428] Setting key 134689544c90467e3cb6e8c7200836a2
I20260924 10:05:07.584088 29494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:05:07 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:07.597344 29494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap starting.
I20260924 10:05:07.601263 29494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:07.602555 29494 log.cc:824] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:07.605947 29494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: No bootstrap required, opened a new log
I20260924 10:05:07.606415 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:07.583925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.13.162.62:45263)
0924 10:05:07.584794 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:07.584809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:07.584859 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:05:07.587033 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:07.587061 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:07.587612 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:07.588093 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:07.589409 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:07.591436 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:07.592815 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:07.593823 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:07.594020 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:07.598066 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:07.598086 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:07.598105 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:07.598151 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:07.601970 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:07.602728 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:07.602737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:07.602745 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:07.602943 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:07.603397 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:07.603400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:07.603402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:07.603971 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:07.604260 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:07.604623 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 10:05:07.605058 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":398,"threads_started":1}
I20260924 10:05:07.613839 29494 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:07.614180 29494 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:07.614252 29494 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0f317e5544467bb59d733201ef0881, State: Initialized, Role: FOLLOWER
I20260924 10:05:07.614799 29494 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:07.615007 29494 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:05:07.615106 29494 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:05:07.615228 29494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:07.617491 29494 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:07.618034 29494 leader_election.cc:302] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0f317e5544467bb59d733201ef0881; no voters:
I20260924 10:05:07.618667 29494 leader_election.cc:288] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:05:07.618801 29499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:05:07.619458 29499 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 LEADER]: Becoming Leader. State: Replica: df0f317e5544467bb59d733201ef0881, State: Running, Role: LEADER
I20260924 10:05:07.620051 29499 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:07.620740 29494 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:05:07.623163 29500 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0f317e5544467bb59d733201ef0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } } }
I20260924 10:05:07.623163 29501 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0f317e5544467bb59d733201ef0881. Latest consensus state: current_term: 1 leader_uuid: "df0f317e5544467bb59d733201ef0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } } }
I20260924 10:05:07.623611 29500 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: This master's current role is: LEADER
I20260924 10:05:07.623610 29501 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: This master's current role is: LEADER
I20260924 10:05:07.627782 29507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:05:07.632942 29507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:05:07.641255 29507 catalog_manager.cc:1347] Generated new cluster ID: 69ddfb780cb8423481ba681d1fb2be43
I20260924 10:05:07.641363 29507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:05:07.652420 29507 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:05:07.653729 29507 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:05:07.662302 29507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Generated new TSK 0
I20260924 10:05:07.663193 29507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:05:07.720132 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:45263
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:05:08.045410 29520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:08.045707 29520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:08.045801 29520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:08.053050 29520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:08.053162 29520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:08.053277 29520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:05:08.063107 29520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:08.064337 29520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:08.065601 29520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:08.077996 29526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:08.078064 29528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:08.078097 29525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:08.078619 29520 server_base.cc:1034] running on GCE node
I20260924 10:05:08.079186 29520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:08.080121 29520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:08.081337 29520 hybrid_clock.cc:648] HybridClock initialized: now 1790244308081292 us; error 41 us; skew 500 ppm
I20260924 10:05:08.081629 29520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:08 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244308, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:08.084829 29520 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:05:08.086431 29520 webserver.cc:460] Webserver started at http://127.13.162.1:39473/ using document root <none> and password file <none>
I20260924 10:05:08.087123 29520 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:08.087230 29520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:08.087515 29520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:05:08.090468 29520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "35ae87923a464e94aebd5d5f70266bab"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "5322439ba72fe68c0f7c7a61eb3b8d22"
server_key_iv: "868d916c78d40ec93ceaa9bf1a775ce8"
server_key_version: "encryptionkey@0"
I20260924 10:05:08.091204 29520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "35ae87923a464e94aebd5d5f70266bab"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "5322439ba72fe68c0f7c7a61eb3b8d22"
server_key_iv: "868d916c78d40ec93ceaa9bf1a775ce8"
server_key_version: "encryptionkey@0"
I20260924 10:05:08.096228 29520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:05:08.099649 29535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:08.101141 29520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:05:08.101282 29520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "35ae87923a464e94aebd5d5f70266bab"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "5322439ba72fe68c0f7c7a61eb3b8d22"
server_key_iv: "868d916c78d40ec93ceaa9bf1a775ce8"
server_key_version: "encryptionkey@0"
I20260924 10:05:08.101471 29520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:08.123189 29520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:08.127328 29520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:08.127583 29520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:08.128497 29520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:08.129777 29520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:08.129849 29520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:08.129942 29520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:08.129984 29520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:08.158129 29520 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:37683
I20260924 10:05:08.158179 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:37683 every 8 connection(s)
I20260924 10:05:08.159384 29520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:05:08.161720 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29520
I20260924 10:05:08.161928 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:05:08.162812 13960 external_mini_cluster.cc:1428] Setting key 790869b18d05cca62556504bc111a708
Sep 24 10:05:08 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244308, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:08.182264 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:08.162254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:47217 (local address 127.13.162.62:45263)
0924 10:05:08.166692 (+ 4438us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:08.166699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:08.166720 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:05:08.166985 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:08.166990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:08.167067 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:08.167173 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:08.168655 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.169650 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.171278 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.171874 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.172018 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:08.176054 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:08.176060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:08.176063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:08.176099 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:08.178744 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:08.179590 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:08.179594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:08.179596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:08.179725 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:08.180249 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:08.180253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:08.180256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:08.180570 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:08.180701 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:08.181710 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0924 10:05:08.181969 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4298,"thread_start_us":4196,"threads_started":1}
I20260924 10:05:08.183713 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:08.162637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:47217)
0924 10:05:08.164177 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:08.164242 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:08.166729 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:08.167377 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:08.167386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:08.167914 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:05:08.168492 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:08.168508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.169802 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.169806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:08.171140 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:08.171153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.172122 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.172130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:08.172424 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:08.173136 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:05:08.173172 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:05:08.175778 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:05:08.178876 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:08.178885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:08.178903 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:08.179456 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:08.179824 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:08.179829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:08.179833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:08.180093 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:08.180704 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:05:08.180712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:05:08.181190 (+ 478us) client_negotiation.cc:769] Sending connection context
0924 10:05:08.181564 (+ 374us) client_negotiation.cc:240] Negotiation successful
0924 10:05:08.182194 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"thread_start_us":936,"threads_started":1}
I20260924 10:05:08.185456 29648 heartbeater.cc:347] Connected to a master server at 127.13.162.62:45263
I20260924 10:05:08.185778 29648 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:05:08.186576 29648 heartbeater.cc:511] Master 127.13.162.62:45263 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:05:08.188853 29459 ts_manager.cc:194] Registered new tserver with Master: 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683)
I20260924 10:05:08.190791 29459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:47217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:05:08.213706 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:45263
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:05:08.538404 29654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:08.538724 29654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:08.538868 29654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:08.546382 29654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:08.546510 29654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:08.546675 29654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:05:08.556466 29654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:08.557725 29654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:08.559013 29654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:08.571578 29659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:08.571550 29662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:08.571512 29660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:08.572755 29654 server_base.cc:1034] running on GCE node
I20260924 10:05:08.573357 29654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:08.574301 29654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:08.575531 29654 hybrid_clock.cc:648] HybridClock initialized: now 1790244308575478 us; error 44 us; skew 500 ppm
I20260924 10:05:08.575840 29654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:08 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244308, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:08.579447 29654 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:05:08.587827 29654 webserver.cc:460] Webserver started at http://127.13.162.2:33371/ using document root <none> and password file <none>
I20260924 10:05:08.588577 29654 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:08.588696 29654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:08.588999 29654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:05:08.592206 29654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "6764d3c06fecbee950ff455ed0fbef93"
server_key_iv: "1cd3f48707c1f6308cfc9ec762332101"
server_key_version: "encryptionkey@0"
I20260924 10:05:08.592949 29654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "6764d3c06fecbee950ff455ed0fbef93"
server_key_iv: "1cd3f48707c1f6308cfc9ec762332101"
server_key_version: "encryptionkey@0"
I20260924 10:05:08.598145 29654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:05:08.601763 29669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:08.603212 29654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:08.603361 29654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "6764d3c06fecbee950ff455ed0fbef93"
server_key_iv: "1cd3f48707c1f6308cfc9ec762332101"
server_key_version: "encryptionkey@0"
I20260924 10:05:08.603609 29654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:08.626573 29654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:08.630821 29654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:08.631099 29654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:08.632071 29654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:08.633440 29654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:08.633524 29654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:08.633632 29654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:08.633679 29654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:08.663831 29654 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:45511
I20260924 10:05:08.663899 29781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:45511 every 8 connection(s)
I20260924 10:05:08.665073 29654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:05:08.666000 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29654
I20260924 10:05:08.666182 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:05:08.666729 13960 external_mini_cluster.cc:1428] Setting key 4d4ef9ea45c694c37ad56f74fad1c5b9
Sep 24 10:05:08 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244308, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:08.686621 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:08.667892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:41299 (local address 127.13.162.62:45263)
0924 10:05:08.668105 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:08.668111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:08.669460 (+ 1349us) server_negotiation.cc:407] Connection header received
0924 10:05:08.671829 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:08.671833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:08.671898 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:08.671979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:08.673804 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.674573 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.676524 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.676916 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.677002 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:08.680454 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:08.680461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:08.680465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:08.680505 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:08.683473 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:08.684237 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:08.684242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:08.684245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:08.684417 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:08.684837 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:08.684841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:08.684843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:08.685142 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:08.685284 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:08.686004 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 10:05:08.686247 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:05:08.687739 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:08.668390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:41299)
0924 10:05:08.669287 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:08.669345 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:08.671570 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:08.672234 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:08.672244 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:08.672903 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:05:08.673633 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:08.673662 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.674712 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.674717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:08.676407 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:08.676421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:08.677124 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:08.677130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:08.677365 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:08.678017 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:05:08.678038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:05:08.680218 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:05:08.683640 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:08.683647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:08.683661 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:08.684124 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:08.684510 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:08.684513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:08.684516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:08.684719 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:08.685274 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:05:08.685282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:05:08.685647 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 10:05:08.685890 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 10:05:08.686476 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":366,"threads_started":1}
I20260924 10:05:08.689389 29782 heartbeater.cc:347] Connected to a master server at 127.13.162.62:45263
I20260924 10:05:08.689684 29782 heartbeater.cc:464] Registering TS with master...
I20260924 10:05:08.690454 29782 heartbeater.cc:511] Master 127.13.162.62:45263 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
I20260924 10:05:08.691865 29459 ts_manager.cc:194] Registered new tserver with Master: ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511)
I20260924 10:05:08.692687 29459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:41299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:05:08.721189 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:45263
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:05:09.040505 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:09.040814 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:09.040961 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:09.048367 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:09.048524 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:09.048691 29787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:05:09.058512 29787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:09.059850 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:09.061265 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:09.074110 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:09.074168 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:09.074079 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:09.075119 29787 server_base.cc:1034] running on GCE node
I20260924 10:05:09.075779 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:09.076730 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:09.077943 29787 hybrid_clock.cc:648] HybridClock initialized: now 1790244309077882 us; error 51 us; skew 500 ppm
I20260924 10:05:09.078269 29787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:09 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244309, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:09.081578 29787 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:05:09.083124 29787 webserver.cc:460] Webserver started at http://127.13.162.3:37075/ using document root <none> and password file <none>
I20260924 10:05:09.083849 29787 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:09.083973 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:09.084283 29787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:05:09.087442 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "4ee444dd66464c938c4f3badc3a9f207"
format_stamp: "Formatted at 2026-09-24 10:05:09 on dist-test-slave-jvsj"
server_key: "3b6a39fe7b97de9773422d4df63793a1"
server_key_iv: "db86a13d7e7516f1243a67e9dc51e4f1"
server_key_version: "encryptionkey@0"
I20260924 10:05:09.088233 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "4ee444dd66464c938c4f3badc3a9f207"
format_stamp: "Formatted at 2026-09-24 10:05:09 on dist-test-slave-jvsj"
server_key: "3b6a39fe7b97de9773422d4df63793a1"
server_key_iv: "db86a13d7e7516f1243a67e9dc51e4f1"
server_key_version: "encryptionkey@0"
I20260924 10:05:09.093500 29787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:05:09.096957 29802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:09.098518 29787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:09.098704 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "4ee444dd66464c938c4f3badc3a9f207"
format_stamp: "Formatted at 2026-09-24 10:05:09 on dist-test-slave-jvsj"
server_key: "3b6a39fe7b97de9773422d4df63793a1"
server_key_iv: "db86a13d7e7516f1243a67e9dc51e4f1"
server_key_version: "encryptionkey@0"
I20260924 10:05:09.098919 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:09.125382 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:09.129693 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:09.129956 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:09.130956 29787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:09.132323 29787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:09.132393 29787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:09.132499 29787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:09.132555 29787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:09.162501 29787 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:42905
I20260924 10:05:09.162531 29914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:42905 every 8 connection(s)
I20260924 10:05:09.164172 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:05:09.172354 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29787
I20260924 10:05:09.172525 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:05:09.173128 13960 external_mini_cluster.cc:1428] Setting key 114013d451bdf4bd59680767dc1db98b
Sep 24 10:05:09 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244309, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:09.183629 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.166735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:55351 (local address 127.13.162.62:45263)
0924 10:05:09.166957 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.166964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.168129 (+ 1165us) server_negotiation.cc:407] Connection header received
0924 10:05:09.169935 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.169940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.170007 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.170095 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:09.171649 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.172400 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.174368 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.174939 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.175065 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.178354 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:09.178360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:09.178364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:09.178396 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:09.180910 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.181624 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.181628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.181630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.181757 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.182109 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.182113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.182116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.182428 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:09.182546 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.183193 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.183361 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:05:09.184612 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.167262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:55351)
0924 10:05:09.167955 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.167993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.169687 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.170293 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.170301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.170939 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:05:09.171497 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.171514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.172613 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.172618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.174204 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.174217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.175154 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.175161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.175385 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.176087 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:05:09.176117 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:05:09.178140 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:05:09.181056 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:09.181064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:09.181078 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:09.181516 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:09.181834 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:09.181838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:09.181840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:09.182016 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:09.182541 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:05:09.182549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:05:09.182884 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.183130 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.183512 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":263,"threads_started":1}
I20260924 10:05:09.186017 29915 heartbeater.cc:347] Connected to a master server at 127.13.162.62:45263
I20260924 10:05:09.186362 29915 heartbeater.cc:464] Registering TS with master...
I20260924 10:05:09.187155 29915 heartbeater.cc:511] Master 127.13.162.62:45263 requested a full tablet report, sending...
I20260924 10:05:09.188602 29459 ts_manager.cc:194] Registered new tserver with Master: 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905)
I20260924 10:05:09.189347 29459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:55351
I20260924 10:05:09.193706 29648 heartbeater.cc:503] Master 127.13.162.62:45263 was elected leader, sending a full tablet report...
I20260924 10:05:09.200368 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:05:09.216709 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.205085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.13.162.62:45263)
0924 10:05:09.205283 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.205287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.205685 (+ 398us) server_negotiation.cc:407] Connection header received
0924 10:05:09.205811 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.205815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.205883 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.205972 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:09.207390 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.208238 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.209514 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.209934 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.210030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.212250 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:09.212255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:09.212258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:09.212293 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:09.214611 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.215177 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.215181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.215183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.215302 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.215575 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.215578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.215580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.215869 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:09.216043 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.216257 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.216425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:05:09.220997 29459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:05:09.227849 29459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:05:09.253717 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.239306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45511 (local address 127.0.0.1:36472)
0924 10:05:09.240704 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.240748 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.240938 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.242050 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.242059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.242090 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.242549 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.242562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.245827 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.245833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.249763 (+ 3930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.249796 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.252773 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.252779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.252986 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.253036 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.253167 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.253358 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":443,"threads_started":1}
I20260924 10:05:09.253734 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.239543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.13.162.2:45511)
0924 10:05:09.240391 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.240399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.241138 (+ 739us) server_negotiation.cc:407] Connection header received
0924 10:05:09.241254 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.241263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.241551 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.241755 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.243769 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.245617 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.250020 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.252582 (+ 2562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.252733 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.252948 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.253225 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.253398 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":559,"threads_started":1}
I20260924 10:05:09.258225 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.240551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:42905 (local address 127.0.0.1:36966)
0924 10:05:09.241070 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.241087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.241201 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.251686 (+ 10485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.251690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.251717 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.252058 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.252069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.253940 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.253945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.256003 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.256017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.257377 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.257383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.257540 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.257572 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.257673 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.257807 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":55,"thread_start_us":268,"threads_started":1}
I20260924 10:05:09.258538 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.241569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.13.162.1:37683)
0924 10:05:09.251422 (+ 9853us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.251430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.251456 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:05:09.251555 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.251563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.251792 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.252020 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.252607 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.254003 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.256227 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.257536 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.257669 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.257900 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.258042 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.258213 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9679,"thread_start_us":9541,"threads_started":1}
I20260924 10:05:09.258559 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.241129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:52458)
0924 10:05:09.250064 (+ 8935us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.250085 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.250228 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.252042 (+ 1814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.252046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.252069 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.252443 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.252455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.254178 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.254183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.256012 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.256031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.257719 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.257726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.257874 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.257905 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.257994 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.258106 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8758,"mutex_wait_us":7,"thread_start_us":9212,"threads_started":1}
I20260924 10:05:09.259068 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.240862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.13.162.3:42905)
0924 10:05:09.251046 (+ 10184us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.251057 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.251097 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:05:09.251199 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.251207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.251487 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.251703 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.252299 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.253782 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.256188 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.257192 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.257306 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.257524 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.258571 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.258755 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9996,"thread_start_us":9839,"threads_started":1}
I20260924 10:05:09.258196 29717 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd83cfbaf9e4192a905fa942bec0c3c (DEFAULT_TABLE table=test-table [id=e87a72838abc45c580911a32a37a7145]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:05:09.260248 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd83cfbaf9e4192a905fa942bec0c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:05:09.262552 29583 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd83cfbaf9e4192a905fa942bec0c3c (DEFAULT_TABLE table=test-table [id=e87a72838abc45c580911a32a37a7145]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:05:09.262739 29850 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd83cfbaf9e4192a905fa942bec0c3c (DEFAULT_TABLE table=test-table [id=e87a72838abc45c580911a32a37a7145]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:05:09.264055 29583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd83cfbaf9e4192a905fa942bec0c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:05:09.264082 29850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd83cfbaf9e4192a905fa942bec0c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:05:09.272006 29933 tablet_bootstrap.cc:492] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Bootstrap starting.
I20260924 10:05:09.275290 29933 tablet_bootstrap.cc:654] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:09.276290 29933 log.cc:824] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:09.278301 29936 tablet_bootstrap.cc:492] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Bootstrap starting.
I20260924 10:05:09.280813 29933 tablet_bootstrap.cc:492] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: No bootstrap required, opened a new log
I20260924 10:05:09.281117 29933 ts_tablet_manager.cc:1397] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:05:09.282013 29936 tablet_bootstrap.cc:654] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:09.283028 29935 tablet_bootstrap.cc:492] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Bootstrap starting.
I20260924 10:05:09.283335 29936 log.cc:824] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:09.286628 29935 tablet_bootstrap.cc:654] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:09.286733 29936 tablet_bootstrap.cc:492] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: No bootstrap required, opened a new log
I20260924 10:05:09.287009 29936 ts_tablet_manager.cc:1397] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 10:05:09.287704 29935 log.cc:824] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:09.290656 29935 tablet_bootstrap.cc:492] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: No bootstrap required, opened a new log
I20260924 10:05:09.290884 29935 ts_tablet_manager.cc:1397] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:05:09.291507 29933 raft_consensus.cc:348] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.292016 29933 raft_consensus.cc:374] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:09.292122 29933 raft_consensus.cc:729] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3cd912c7024ae8ab9aa6b0a6383489, State: Initialized, Role: FOLLOWER
I20260924 10:05:09.292779 29933 consensus_queue.cc:260] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.293912 29782 heartbeater.cc:503] Master 127.13.162.62:45263 was elected leader, sending a full tablet report...
I20260924 10:05:09.294655 29933 ts_tablet_manager.cc:1428] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 10:05:09.296819 29936 raft_consensus.cc:348] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.297255 29936 raft_consensus.cc:374] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:09.297360 29936 raft_consensus.cc:729] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee444dd66464c938c4f3badc3a9f207, State: Initialized, Role: FOLLOWER
I20260924 10:05:09.297996 29936 consensus_queue.cc:260] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.299113 29915 heartbeater.cc:503] Master 127.13.162.62:45263 was elected leader, sending a full tablet report...
I20260924 10:05:09.299048 29935 raft_consensus.cc:348] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.299492 29935 raft_consensus.cc:374] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:09.299592 29935 raft_consensus.cc:729] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae87923a464e94aebd5d5f70266bab, State: Initialized, Role: FOLLOWER
I20260924 10:05:09.299875 29936 ts_tablet_manager.cc:1428] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 10:05:09.300246 29935 consensus_queue.cc:260] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.302664 29935 ts_tablet_manager.cc:1428] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 10:05:09.383709 29940 raft_consensus.cc:482] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:05:09.384174 29940 raft_consensus.cc:504] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.384752 29941 raft_consensus.cc:482] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:05:09.385022 29941 raft_consensus.cc:504] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.386461 29940 leader_election.cc:288] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511), 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683)
I20260924 10:05:09.386963 29941 leader_election.cc:288] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511), 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905)
I20260924 10:05:09.395045 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.386807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45511 (local address 127.13.162.3:34113)
0924 10:05:09.387560 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.387578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.387697 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.388229 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.388234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.388271 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.388473 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.388484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.390074 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.390080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.392267 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.392285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.394169 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.394175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.394407 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.394446 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.394552 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.394717 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630}
I20260924 10:05:09.395823 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.386818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:34113 (local address 127.13.162.2:45511)
0924 10:05:09.387036 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.387042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.387784 (+ 742us) server_negotiation.cc:407] Connection header received
0924 10:05:09.387872 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.387877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.387954 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.388058 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.388669 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.389872 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.392480 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.393838 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.393992 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.394072 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.395378 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.395533 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:05:09.401109 29737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" is_pre_election: true
I20260924 10:05:09.401763 29737 raft_consensus.cc:2454] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee444dd66464c938c4f3badc3a9f207 in term 0.
I20260924 10:05:09.402812 29806 leader_election.cc:302] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee444dd66464c938c4f3badc3a9f207, ca3cd912c7024ae8ab9aa6b0a6383489; no voters:
I20260924 10:05:09.403389 29940 raft_consensus.cc:2781] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:05:09.403551 29940 raft_consensus.cc:482] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:05:09.403661 29940 raft_consensus.cc:3037] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:09.405547 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.387500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:57507 (local address 127.13.162.1:37683)
0924 10:05:09.387680 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.387686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.396314 (+ 8628us) server_negotiation.cc:407] Connection header received
0924 10:05:09.397220 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.397226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.397304 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.397363 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.397941 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.399169 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.402344 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.404111 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.404278 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.404359 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.405138 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.405284 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":62}
I20260924 10:05:09.406987 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.387196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.13.162.3:57507)
0924 10:05:09.396212 (+ 9016us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.396233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.397047 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.397549 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.397554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.397581 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.397786 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.397796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.399365 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.399370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.402142 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.402156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.404485 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.404495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.404768 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.404801 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.404892 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.405826 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8905,"thread_start_us":12673,"threads_started":1}
I20260924 10:05:09.407483 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.387508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:58159 (local address 127.13.162.2:45511)
0924 10:05:09.396624 (+ 9116us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.396630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.396645 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:05:09.396704 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.396709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.396787 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.396872 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.400943 (+ 4071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.402114 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.405374 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.406562 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.406729 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.406804 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.407086 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.407223 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9004,"thread_start_us":12657,"threads_started":1}
I20260924 10:05:09.407898 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.387286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45511 (local address 127.13.162.1:58159)
0924 10:05:09.388053 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.388074 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.388196 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.400346 (+ 12150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.400351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.400381 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.400773 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.400788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.402281 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.402286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.405149 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.405165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.406772 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.406778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.406937 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.406960 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.407037 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.407344 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":55,"thread_start_us":648,"threads_started":1}
I20260924 10:05:09.408218 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.400130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:48933 (local address 127.13.162.3:42905)
0924 10:05:09.400316 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.400321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.400337 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:05:09.400393 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.400397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.400459 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.400536 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:09.401306 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.402408 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.405344 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.406728 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.406866 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.406931 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.407863 (+ 932us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.408005 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:05:09.408401 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.388374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:42905 (local address 127.13.162.1:48933)
0924 10:05:09.399763 (+ 11389us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:09.399783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:09.399917 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:09.400678 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:09.400687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:09.400720 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:09.401108 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.401130 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.402643 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.402653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.405155 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:09.405165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.407362 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.407369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:09.407538 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.407571 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:05:09.407813 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 10:05:09.407948 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11261,"mutex_wait_us":107,"thread_start_us":11605,"threads_started":1}
I20260924 10:05:09.408353 29603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae87923a464e94aebd5d5f70266bab" is_pre_election: true
I20260924 10:05:09.407351 29940 raft_consensus.cc:504] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.408887 29603 raft_consensus.cc:2454] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee444dd66464c938c4f3badc3a9f207 in term 0.
I20260924 10:05:09.409134 29737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "35ae87923a464e94aebd5d5f70266bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" is_pre_election: true
I20260924 10:05:09.409365 29737 raft_consensus.cc:2454] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae87923a464e94aebd5d5f70266bab in term 0.
I20260924 10:05:09.409318 29870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "35ae87923a464e94aebd5d5f70266bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee444dd66464c938c4f3badc3a9f207" is_pre_election: true
I20260924 10:05:09.409811 29940 leader_election.cc:288] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 election: Requested vote from peers ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511), 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683)
I20260924 10:05:09.409893 29870 raft_consensus.cc:2379] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35ae87923a464e94aebd5d5f70266bab in current term 1: Already voted for candidate 4ee444dd66464c938c4f3badc3a9f207 in this term.
I20260924 10:05:09.409883 29539 leader_election.cc:302] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ae87923a464e94aebd5d5f70266bab, ca3cd912c7024ae8ab9aa6b0a6383489; no voters:
I20260924 10:05:09.410305 29941 raft_consensus.cc:2781] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:05:09.410281 29737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
I20260924 10:05:09.410492 29941 raft_consensus.cc:482] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:05:09.410501 29737 raft_consensus.cc:3037] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:09.410614 29941 raft_consensus.cc:3037] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:09.410580 29603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae87923a464e94aebd5d5f70266bab"
W20260924 10:05:09.411585 29649 tablet.cc:2369] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:05:09.412838 29737 raft_consensus.cc:2454] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee444dd66464c938c4f3badc3a9f207 in term 1.
I20260924 10:05:09.413254 29806 leader_election.cc:302] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee444dd66464c938c4f3badc3a9f207, ca3cd912c7024ae8ab9aa6b0a6383489; no voters:
I20260924 10:05:09.413497 29941 raft_consensus.cc:504] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:09.413985 29603 raft_consensus.cc:2379] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ee444dd66464c938c4f3badc3a9f207 in current term 1: Already voted for candidate 35ae87923a464e94aebd5d5f70266bab in this term.
I20260924 10:05:09.414192 29941 leader_election.cc:288] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [CANDIDATE]: Term 1 election: Requested vote from peers ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511), 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905)
I20260924 10:05:09.414357 29940 raft_consensus.cc:2781] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:05:09.414688 29737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "35ae87923a464e94aebd5d5f70266bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
I20260924 10:05:09.414815 29870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd83cfbaf9e4192a905fa942bec0c3c" candidate_uuid: "35ae87923a464e94aebd5d5f70266bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee444dd66464c938c4f3badc3a9f207"
I20260924 10:05:09.414974 29737 raft_consensus.cc:2379] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 35ae87923a464e94aebd5d5f70266bab in current term 1: Already voted for candidate 4ee444dd66464c938c4f3badc3a9f207 in this term.
I20260924 10:05:09.415863 29940 raft_consensus.cc:686] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee444dd66464c938c4f3badc3a9f207, State: Running, Role: LEADER
W20260924 10:05:09.415881 29916 tablet.cc:2369] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:05:09.416363 29940 consensus_queue.cc:237] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
W20260924 10:05:09.417016 29783 tablet.cc:2369] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:05:09.419727 29539 leader_election.cc:302] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35ae87923a464e94aebd5d5f70266bab; no voters: 4ee444dd66464c938c4f3badc3a9f207, ca3cd912c7024ae8ab9aa6b0a6383489
I20260924 10:05:09.419986 29941 raft_consensus.cc:2726] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 10:05:09.421423 29458 catalog_manager.cc:5495] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "4ee444dd66464c938c4f3badc3a9f207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } health_report { overall_health: UNKNOWN } } }
I20260924 10:05:09.442175 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.436777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.13.162.3:42905)
0924 10:05:09.436953 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.436959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.436977 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:05:09.437149 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.437153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.437242 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.437365 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:05:09.437739 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.438539 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.440013 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.440482 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.440589 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.441077 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:05:09.441583 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:05:09.441713 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.441807 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.441892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:05:09.450263 29737 raft_consensus.cc:1261] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Refusing update from remote peer 4ee444dd66464c938c4f3badc3a9f207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:05:09.450263 29603 raft_consensus.cc:1261] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Refusing update from remote peer 4ee444dd66464c938c4f3badc3a9f207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:05:09.451110 29946 consensus_queue.cc:1035] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:05:09.451455 29940 consensus_queue.cc:1035] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:05:09.461901 29949 mvcc.cc:205] Tried to move back new op lower bound from 7332840691498823680 to 7332840691373080576. Current Snapshot: MvccSnapshot[applied={T|T < 7332840691498823680}]
I20260924 10:05:09.466279 29951 mvcc.cc:205] Tried to move back new op lower bound from 7332840691498823680 to 7332840691373080576. Current Snapshot: MvccSnapshot[applied={T|T < 7332840691498823680}]
I20260924 10:05:09.479534 29458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:05:09.479794 29458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:05:09.483402 29458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Sending DeleteTablet for 3 replicas of tablet 3fd83cfbaf9e4192a905fa942bec0c3c
I20260924 10:05:09.484614 29717 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd83cfbaf9e4192a905fa942bec0c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:05:09 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:36472
I20260924 10:05:09.484920 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29520
I20260924 10:05:09.484974 29583 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd83cfbaf9e4192a905fa942bec0c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:05:09 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:52458
I20260924 10:05:09.485385 29850 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd83cfbaf9e4192a905fa942bec0c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:05:09 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:36966
I20260924 10:05:09.509639 29961 tablet_replica.cc:330] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: stopping tablet replica
I20260924 10:05:09.510392 29961 raft_consensus.cc:2229] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:05:09.511263 29961 raft_consensus.cc:2258] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:05:09.511978 29959 tablet_replica.cc:330] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: stopping tablet replica
I20260924 10:05:09.512758 29959 raft_consensus.cc:2229] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:05:09.514968 29961 ts_tablet_manager.cc:1905] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:09.515586 29959 raft_consensus.cc:2258] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:05:09.518944 29959 ts_tablet_manager.cc:1905] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:09.521907 29960 tablet_replica.cc:330] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: stopping tablet replica
I20260924 10:05:09.522457 29960 raft_consensus.cc:2229] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:05:09.523021 29960 raft_consensus.cc:2258] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:05:09.524679 29961 ts_tablet_manager.cc:1918] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:05:09.524876 29961 log.cc:1196] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/3fd83cfbaf9e4192a905fa942bec0c3c
I20260924 10:05:09.525408 29961 ts_tablet_manager.cc:1939] T 3fd83cfbaf9e4192a905fa942bec0c3c P 4ee444dd66464c938c4f3badc3a9f207: Deleting consensus metadata
I20260924 10:05:09.526945 29959 ts_tablet_manager.cc:1918] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:05:09.527199 29959 log.cc:1196] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/3fd83cfbaf9e4192a905fa942bec0c3c
I20260924 10:05:09.527541 29446 catalog_manager.cc:4841] TS 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905): tablet 3fd83cfbaf9e4192a905fa942bec0c3c (table test-table [id=e87a72838abc45c580911a32a37a7145]) successfully deleted
I20260924 10:05:09.527657 29960 ts_tablet_manager.cc:1905] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:09.527925 29959 ts_tablet_manager.cc:1939] T 3fd83cfbaf9e4192a905fa942bec0c3c P ca3cd912c7024ae8ab9aa6b0a6383489: Deleting consensus metadata
I20260924 10:05:09.529764 29446 catalog_manager.cc:4841] TS ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511): tablet 3fd83cfbaf9e4192a905fa942bec0c3c (table test-table [id=e87a72838abc45c580911a32a37a7145]) successfully deleted
Sep 24 10:05:09 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:05:09.533759 29960 ts_tablet_manager.cc:1918] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:05:09.533952 29960 log.cc:1196] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/3fd83cfbaf9e4192a905fa942bec0c3c
I20260924 10:05:09.534456 29960 ts_tablet_manager.cc:1939] T 3fd83cfbaf9e4192a905fa942bec0c3c P 35ae87923a464e94aebd5d5f70266bab: Deleting consensus metadata
I20260924 10:05:09.536391 29446 catalog_manager.cc:4841] TS 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683): tablet 3fd83cfbaf9e4192a905fa942bec0c3c (table test-table [id=e87a72838abc45c580911a32a37a7145]) successfully deleted
I20260924 10:05:09.540266 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.486106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.13.162.1:37683)
0924 10:05:09.486292 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.486298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.516295 (+ 29997us) server_negotiation.cc:407] Connection header received
0924 10:05:09.516506 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.516511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.516631 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.516734 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:09.518979 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.520050 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.522737 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.523489 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.523609 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.533279 (+ 9670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:09.533294 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:09.533314 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:09.533365 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:09.537086 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.538001 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.538009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.538011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.538172 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.538674 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.538680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.538684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.539081 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:09.539312 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.539728 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.539920 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:05:09.540874 29643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:09.647390 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29520
I20260924 10:05:09.662242 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29654
Sep 24 10:05:09 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:05:09.681535 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.663044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.13.162.2:45511)
0924 10:05:09.663273 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.663279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.663300 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:05:09.663527 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.663533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.663643 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.663743 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:09.665471 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.666641 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.668573 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.669200 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.669327 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.674621 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:09.674641 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:09.674657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:09.674710 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:09.678514 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.679308 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.679318 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.679321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.679458 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.679846 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.679850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.679854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.680260 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:09.680554 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.680811 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.681023 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 10:05:09.682147 29777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:09.828426 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29654
I20260924 10:05:09.842229 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29787
Sep 24 10:05:09 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:05:09.859453 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:09.842994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.13.162.3:42905)
0924 10:05:09.843241 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:09.843247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:09.843268 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:05:09.843467 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:09.843471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:09.843543 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:09.843625 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:09.844908 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.845720 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.847231 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:09.847677 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:09.847807 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:09.852208 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:09.852225 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:09.852245 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:09.852296 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:09.856339 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.857103 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.857117 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.857121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.857289 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:09.857660 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:09.857665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:09.857669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:09.858203 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:09.858549 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:09.858837 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 10:05:09.859087 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:05:09.860363 29910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:09.983270 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29787
I20260924 10:05:10.000177 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29427
I20260924 10:05:10.001312 29489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:10.109776 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29427
I20260924 10:05:30.127506 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45263
--webserver_interface=127.13.162.62
--webserver_port=40591
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:05:30.459379 29967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:30.459695 29967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:30.459826 29967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:30.469586 29967 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:05:30.469704 29967 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:05:30.469769 29967 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:05:30.469842 29967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:05:30.469913 29967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:30.470005 29967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:30.470072 29967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:05:30.470142 29967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:05:30.480574 29967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=40591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:30.482024 29967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:30.483513 29967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:30.506266 29974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:30.506409 29973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:30.506356 29976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:30.506578 29967 server_base.cc:1034] running on GCE node
I20260924 10:05:30.507788 29967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:30.510070 29967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:30.511400 29967 hybrid_clock.cc:648] HybridClock initialized: now 1790244330511299 us; error 66 us; skew 500 ppm
I20260924 10:05:30.511809 29967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:30 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244330, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:30.520866 29967 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:05:30.522768 29967 webserver.cc:460] Webserver started at http://127.13.162.62:40591/ using document root <none> and password file <none>
I20260924 10:05:30.523584 29967 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:30.523721 29967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:30.530297 29967 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 10:05:30.533881 29983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:30.535693 29967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 10:05:30.535915 29967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "df0f317e5544467bb59d733201ef0881"
format_stamp: "Formatted at 2026-09-24 10:05:07 on dist-test-slave-jvsj"
server_key: "396ca37e66ba6c54169cc2ed0a221c88"
server_key_iv: "59ffd1422a802d8c511b3f5bc5746b58"
server_key_version: "encryptionkey@0"
I20260924 10:05:30.537082 29967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:30.556015 29967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:30.560316 29967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:30.560595 29967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:30.579355 29967 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:45263
I20260924 10:05:30.579381 30034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:45263 every 8 connection(s)
I20260924 10:05:30.581027 29967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:05:30.588771 30035 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:05:30.590582 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 29967
I20260924 10:05:30.591380 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:37683
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:45263
--webserver_port=39473
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:05:30.592481 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap starting.
I20260924 10:05:30.613049 30035 log.cc:824] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:30.625907 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:05:30.626489 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap complete.
I20260924 10:05:30.635921 30035 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:30.636797 30035 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0f317e5544467bb59d733201ef0881, State: Initialized, Role: FOLLOWER
I20260924 10:05:30.637454 30035 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:30.637699 30035 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:05:30.637815 30035 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:05:30.637971 30035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:05:30.642298 30035 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:30.642895 30035 leader_election.cc:302] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0f317e5544467bb59d733201ef0881; no voters:
I20260924 10:05:30.643527 30035 leader_election.cc:288] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:05:30.643682 30039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:05:30.644765 30039 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 LEADER]: Becoming Leader. State: Replica: df0f317e5544467bb59d733201ef0881, State: Running, Role: LEADER
I20260924 10:05:30.645421 30039 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:30.645845 30035 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:05:30.647943 30041 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0f317e5544467bb59d733201ef0881. Latest consensus state: current_term: 2 leader_uuid: "df0f317e5544467bb59d733201ef0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } } }
I20260924 10:05:30.648310 30041 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: This master's current role is: LEADER
I20260924 10:05:30.648713 30040 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df0f317e5544467bb59d733201ef0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } } }
I20260924 10:05:30.649080 30040 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: This master's current role is: LEADER
I20260924 10:05:30.650234 30044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:05:30.659511 30044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:05:30.661290 30044 catalog_manager.cc:1259] Loaded cluster ID: 69ddfb780cb8423481ba681d1fb2be43
I20260924 10:05:30.661497 30044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:05:30.665973 30044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:05:30.667829 30044 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Loaded TSK: 0
I20260924 10:05:30.707019 30044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Generated new TSK 1
I20260924 10:05:30.707475 30044 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:05:30.956655 30037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:30.956992 30037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:30.957122 30037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:30.964453 30037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:30.964588 30037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:30.964751 30037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:05:30.975114 30037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:37683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=39473
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:30.976449 30037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:30.977739 30037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:30.993096 30063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:30.994614 30062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:30.998984 30065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:30.999655 30037 server_base.cc:1034] running on GCE node
I20260924 10:05:31.000324 30037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:31.001324 30037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:31.002532 30037 hybrid_clock.cc:648] HybridClock initialized: now 1790244331002479 us; error 38 us; skew 500 ppm
I20260924 10:05:31.002862 30037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:31 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244331, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:31.006022 30037 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:05:31.007678 30037 webserver.cc:460] Webserver started at http://127.13.162.1:39473/ using document root <none> and password file <none>
I20260924 10:05:31.008392 30037 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:31.008517 30037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:31.014302 30037 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:05:31.017290 30072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.018870 30037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:31.019029 30037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "35ae87923a464e94aebd5d5f70266bab"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "5322439ba72fe68c0f7c7a61eb3b8d22"
server_key_iv: "868d916c78d40ec93ceaa9bf1a775ce8"
server_key_version: "encryptionkey@0"
I20260924 10:05:31.020031 30037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:31.031220 30037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:31.035513 30037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:31.035768 30037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:31.036816 30037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:31.038122 30037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:31.038210 30037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.038298 30037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:31.038385 30037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 10:05:31.069070 30037 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:37683
I20260924 10:05:31.069155 30184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:37683 every 8 connection(s)
I20260924 10:05:31.071007 30037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:05:31.074232 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30037
I20260924 10:05:31.074944 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:45511
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:45263
--webserver_port=33371
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:05:31 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244331, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:31.096864 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:31.073913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:59319 (local address 127.13.162.62:45263)
0924 10:05:31.074578 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:31.074595 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:31.075246 (+ 651us) server_negotiation.cc:407] Connection header received
0924 10:05:31.077468 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:31.077486 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:31.077991 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:31.078441 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:31.079976 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.081788 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.084133 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.084846 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.084998 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:31.088945 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:31.088953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:31.088971 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:31.089015 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:31.091899 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:31.092806 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:31.092814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:31.092821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:31.093004 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:31.093501 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:31.093504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:31.093506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:31.094046 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:31.094233 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:31.095159 (+ 926us) server_negotiation.cc:299] Negotiation successful
0924 10:05:31.095592 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":254,"threads_started":1}
I20260924 10:05:31.096864 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:31.074140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:59319)
0924 10:05:31.075018 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:31.075077 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:31.076808 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:31.078642 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:31.078651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:31.079191 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:05:31.079778 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:31.079797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.081995 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.081999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:31.083953 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:31.083971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.085088 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.085095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:31.085367 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:31.086085 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:05:31.086112 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:05:31.088639 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:05:31.092087 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:31.092099 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:31.092117 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:31.092634 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:31.093135 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:31.093141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:31.093145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:31.093378 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:31.094207 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:05:31.094217 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:05:31.094629 (+ 412us) client_negotiation.cc:769] Sending connection context
0924 10:05:31.094878 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 10:05:31.095507 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":312,"threads_started":1}
I20260924 10:05:31.099789 30185 heartbeater.cc:347] Connected to a master server at 127.13.162.62:45263
I20260924 10:05:31.100099 30185 heartbeater.cc:464] Registering TS with master...
I20260924 10:05:31.100814 30185 heartbeater.cc:511] Master 127.13.162.62:45263 requested a full tablet report, sending...
I20260924 10:05:31.103036 30000 ts_manager.cc:194] Registered new tserver with Master: 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683)
I20260924 10:05:31.104565 30000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:59319
W20260924 10:05:31.407840 30189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:31.408164 30189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:31.408314 30189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:31.415882 30189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:31.416020 30189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:31.416134 30189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:05:31.426479 30189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:45511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=33371
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:31.427729 30189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:31.429059 30189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:31.441629 30194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:31.441601 30195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:31.441869 30197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:31.442482 30189 server_base.cc:1034] running on GCE node
I20260924 10:05:31.443092 30189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:31.444038 30189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:31.445266 30189 hybrid_clock.cc:648] HybridClock initialized: now 1790244331445194 us; error 66 us; skew 500 ppm
I20260924 10:05:31.445569 30189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:31 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244331, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:31.448691 30189 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:05:31.450150 30189 webserver.cc:460] Webserver started at http://127.13.162.2:33371/ using document root <none> and password file <none>
I20260924 10:05:31.450937 30189 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:31.451053 30189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:31.456672 30189 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:05:31.459445 30204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.460880 30189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:31.461050 30189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "6764d3c06fecbee950ff455ed0fbef93"
server_key_iv: "1cd3f48707c1f6308cfc9ec762332101"
server_key_version: "encryptionkey@0"
I20260924 10:05:31.462030 30189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:31.481670 30189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:31.485833 30189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:31.486081 30189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:31.487105 30189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:31.488410 30189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:31.488492 30189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.488605 30189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:31.488672 30189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.519138 30189 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:45511
I20260924 10:05:31.519191 30316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:45511 every 8 connection(s)
I20260924 10:05:31.520623 30189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:05:31.530797 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30189
I20260924 10:05:31.531567 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:42905
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:45263
--webserver_port=37075
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:05:31 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244331, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:31.541505 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:31.523022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:36101 (local address 127.13.162.62:45263)
0924 10:05:31.523176 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:31.523180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:31.524380 (+ 1200us) server_negotiation.cc:407] Connection header received
0924 10:05:31.526206 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:31.526221 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:31.526303 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:31.526429 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:31.527916 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.528612 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.530125 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.530727 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.530859 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:31.535475 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:31.535481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:31.535484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:31.535526 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:31.537982 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:31.538985 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:31.538991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:31.538995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:31.539148 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:31.539604 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:31.539608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:31.539611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:31.539986 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:31.540131 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:31.540887 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 10:05:31.541124 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:05:31.542454 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:31.523542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:36101)
0924 10:05:31.524209 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:31.524255 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:31.525937 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:31.526665 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:31.526673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:31.527194 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:05:31.527753 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:31.527771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.528770 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.528774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:31.529988 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:31.530000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:31.530983 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:31.530991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:31.531337 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:31.532120 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:05:31.532151 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:05:31.535140 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:05:31.538150 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:31.538162 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:31.538184 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:31.538857 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:31.539318 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:31.539321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:31.539324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:31.539511 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:31.540090 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:05:31.540095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:05:31.540531 (+ 436us) client_negotiation.cc:769] Sending connection context
0924 10:05:31.540768 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 10:05:31.541164 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
I20260924 10:05:31.543836 30317 heartbeater.cc:347] Connected to a master server at 127.13.162.62:45263
I20260924 10:05:31.544184 30317 heartbeater.cc:464] Registering TS with master...
I20260924 10:05:31.545015 30317 heartbeater.cc:511] Master 127.13.162.62:45263 requested a full tablet report, sending...
I20260924 10:05:31.546623 30000 ts_manager.cc:194] Registered new tserver with Master: ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511)
I20260924 10:05:31.547480 30000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:36101
W20260924 10:05:31.863413 30320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:31.863695 30320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:31.863865 30320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:31.871156 30320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:31.871277 30320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:31.871421 30320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:05:31.881237 30320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:42905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=37075
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:31.882561 30320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:31.883838 30320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:31.896533 30325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:31.896624 30328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:31.896576 30326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:31.897398 30320 server_base.cc:1034] running on GCE node
I20260924 10:05:31.897990 30320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:31.898998 30320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:31.900202 30320 hybrid_clock.cc:648] HybridClock initialized: now 1790244331900154 us; error 38 us; skew 500 ppm
I20260924 10:05:31.900537 30320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:31 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244331, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:31.903913 30320 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:05:31.905308 30320 webserver.cc:460] Webserver started at http://127.13.162.3:37075/ using document root <none> and password file <none>
I20260924 10:05:31.906010 30320 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:31.906140 30320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:31.911523 30320 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:05:31.914384 30335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.915747 30320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:31.915902 30320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "4ee444dd66464c938c4f3badc3a9f207"
format_stamp: "Formatted at 2026-09-24 10:05:09 on dist-test-slave-jvsj"
server_key: "3b6a39fe7b97de9773422d4df63793a1"
server_key_iv: "db86a13d7e7516f1243a67e9dc51e4f1"
server_key_version: "encryptionkey@0"
I20260924 10:05:31.916887 30320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:31.960078 30320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:31.964164 30320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:31.964419 30320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:31.965492 30320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:31.966943 30320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:31.967037 30320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.967139 30320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:31.967214 30320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:31.997298 30320 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:42905
I20260924 10:05:31.997370 30447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:42905 every 8 connection(s)
I20260924 10:05:31.998795 30320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:05:32.007982 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30320
Sep 24 10:05:32 dist-test-slave-jvsj krb5kdc[29414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244331, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:05:32.020773 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.001417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:37713 (local address 127.13.162.62:45263)
0924 10:05:32.001593 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.001598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.002844 (+ 1246us) server_negotiation.cc:407] Connection header received
0924 10:05:32.004699 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.004703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.004779 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.004872 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:32.006462 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.007171 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.009339 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.009986 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.010064 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.014268 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:32.014275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:32.014281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:32.014361 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:32.017185 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.018161 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.018165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.018169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.018388 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.018826 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.018831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.018834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.019173 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:32.019367 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.020147 (+ 780us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.020387 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:05:32.021903 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.001866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:37713)
0924 10:05:32.002680 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.002721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.004416 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.005083 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.005091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.005669 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:05:32.006271 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.006289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.007335 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.007339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.009131 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.009146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.010445 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.010450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.010661 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.011314 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:05:32.011337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:05:32.014010 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:05:32.017351 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:32.017362 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:32.017385 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:32.018024 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:32.018499 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:05:32.018502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:05:32.018505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:05:32.018702 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:05:32.019344 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:05:32.019350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:05:32.019704 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.020013 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.020535 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":276,"threads_started":1}
I20260924 10:05:32.023301 30448 heartbeater.cc:347] Connected to a master server at 127.13.162.62:45263
I20260924 10:05:32.023650 30448 heartbeater.cc:464] Registering TS with master...
I20260924 10:05:32.024287 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.009088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.13.162.62:45263)
0924 10:05:32.009950 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.009954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.009967 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:05:32.010007 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.010011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.010065 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.010161 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:32.011796 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.012679 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.014641 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.015266 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.015418 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.018234 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:32.018239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:32.018244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:32.018292 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:32.021309 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.022140 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.022145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.022147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.022280 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.022821 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.022825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.022828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.023121 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:32.023355 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.023695 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.023932 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":562,"threads_started":1}
I20260924 10:05:32.024513 30448 heartbeater.cc:511] Master 127.13.162.62:45263 requested a full tablet report, sending...
I20260924 10:05:32.026055 29999 ts_manager.cc:194] Registered new tserver with Master: 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905)
I20260924 10:05:32.027065 29999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:37713
I20260924 10:05:32.039180 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:05:32.045619 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.040042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.13.162.62:45263)
0924 10:05:32.040201 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.040205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.040640 (+ 435us) server_negotiation.cc:407] Connection header received
0924 10:05:32.040818 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.040822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.041007 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.041134 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:05:32.041483 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.042230 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.043765 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.044180 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.044289 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.044797 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:05:32.045014 (+ 217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 10:05:32.045360 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33578: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 10:05:32.045743 30452 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33578: token expired
W20260924 10:05:32.046281 29920 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.162.62:45263)
I20260924 10:05:32.058748 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.046734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.13.162.62:45263)
0924 10:05:32.046943 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.046948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.046991 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:05:32.047104 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.047107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.047164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.047235 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:32.048591 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.049286 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.050639 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.051077 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.051169 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.053824 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:32.053830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:32.053833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:32.053877 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:32.056354 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.056998 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.057002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.057004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.057128 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.057412 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.057415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.057417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.057734 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:32.057922 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.058260 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.058487 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:05:32.062170 29999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:05:32.069793 29999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:05:32.092660 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.082988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.13.162.3:42905)
0924 10:05:32.083550 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.083560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.083654 (+ 94us) server_negotiation.cc:407] Connection header received
0924 10:05:32.084409 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.084418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.084717 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.084928 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.085687 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.087107 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.089781 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.091306 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.091437 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.091674 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.092140 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.092314 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":324,"threads_started":1}
I20260924 10:05:32.092767 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.082770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:42905 (local address 127.0.0.1:45724)
0924 10:05:32.083541 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.083575 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.083993 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.084973 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.084982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.085027 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.085490 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.085504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.087230 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.087235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.089584 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.089599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.091615 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.091623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.091867 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.091938 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.092078 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.092336 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":34,"thread_start_us":369,"threads_started":1}
I20260924 10:05:32.096009 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.084205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.13.162.2:45511)
0924 10:05:32.085575 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.085584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.086093 (+ 509us) server_negotiation.cc:407] Connection header received
0924 10:05:32.087739 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.087748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.087947 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.088114 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.088789 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.090113 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.093638 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.094937 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.095069 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.095301 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.095539 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.095708 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":1061,"threads_started":1}
I20260924 10:05:32.096032 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.083926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45511 (local address 127.0.0.1:57244)
0924 10:05:32.085996 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.086014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.087547 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.088114 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.088119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.088143 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.088602 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.088614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.090229 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.090234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.093454 (+ 3220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.093470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.095170 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.095177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.095343 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.095379 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.095460 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.095603 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1962,"mutex_wait_us":83,"spinlock_wait_cycles":2816,"thread_start_us":1846,"threads_started":1}
I20260924 10:05:32.097149 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.086477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.13.162.1:37683)
0924 10:05:32.087199 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.087207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.087234 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:05:32.087768 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.087777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.088022 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.088201 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.088998 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.090148 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.094447 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.095948 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.096083 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.096295 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.096789 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.096920 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":822,"threads_started":1}
I20260924 10:05:32.097250 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.086112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50696)
0924 10:05:32.086844 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.086860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.087560 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.088401 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.088406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.088425 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.088849 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.088859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.090343 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.090349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.093740 (+ 3391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.093759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.096439 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.096445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.096610 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.096640 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.096732 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.096871 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":447,"spinlock_wait_cycles":1319808,"thread_start_us":264,"threads_started":1}
I20260924 10:05:32.097096 30383 tablet_service.cc:1467] Processing CreateTablet for tablet 0446de919df5430891441fc691efccee (DEFAULT_TABLE table=test-table [id=70ef68b9b48f47639f44a060e403c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:05:32.099479 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0446de919df5430891441fc691efccee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:05:32.100751 30120 tablet_service.cc:1467] Processing CreateTablet for tablet 0446de919df5430891441fc691efccee (DEFAULT_TABLE table=test-table [id=70ef68b9b48f47639f44a060e403c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:05:32.100955 30250 tablet_service.cc:1467] Processing CreateTablet for tablet 0446de919df5430891441fc691efccee (DEFAULT_TABLE table=test-table [id=70ef68b9b48f47639f44a060e403c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:05:32.102154 30120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0446de919df5430891441fc691efccee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:05:32.102372 30250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0446de919df5430891441fc691efccee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:05:32.107561 30185 heartbeater.cc:503] Master 127.13.162.62:45263 was elected leader, sending a full tablet report...
I20260924 10:05:32.118291 30464 tablet_bootstrap.cc:492] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Bootstrap starting.
I20260924 10:05:32.121260 30465 tablet_bootstrap.cc:492] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Bootstrap starting.
I20260924 10:05:32.122496 30464 tablet_bootstrap.cc:654] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:32.123591 30464 log.cc:824] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:32.125161 30465 tablet_bootstrap.cc:654] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:32.126892 30466 tablet_bootstrap.cc:492] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Bootstrap starting.
I20260924 10:05:32.127959 30465 log.cc:824] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:32.129618 30464 tablet_bootstrap.cc:492] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: No bootstrap required, opened a new log
I20260924 10:05:32.129917 30464 ts_tablet_manager.cc:1397] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260924 10:05:32.131764 30466 tablet_bootstrap.cc:654] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Neither blocks nor log segments found. Creating new log.
I20260924 10:05:32.133234 30466 log.cc:824] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:32.136705 30465 tablet_bootstrap.cc:492] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: No bootstrap required, opened a new log
I20260924 10:05:32.137032 30465 ts_tablet_manager.cc:1397] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260924 10:05:32.137108 30466 tablet_bootstrap.cc:492] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: No bootstrap required, opened a new log
I20260924 10:05:32.137372 30466 ts_tablet_manager.cc:1397] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260924 10:05:32.138018 30464 raft_consensus.cc:348] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.138418 30464 raft_consensus.cc:374] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:32.138526 30464 raft_consensus.cc:729] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae87923a464e94aebd5d5f70266bab, State: Initialized, Role: FOLLOWER
I20260924 10:05:32.139153 30464 consensus_queue.cc:260] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.147632 30464 ts_tablet_manager.cc:1428] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Time spent starting tablet: real 0.018s user 0.008s sys 0.011s
I20260924 10:05:32.147737 30466 raft_consensus.cc:348] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.147737 30465 raft_consensus.cc:348] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.148192 30465 raft_consensus.cc:374] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:32.148192 30466 raft_consensus.cc:374] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:05:32.148296 30466 raft_consensus.cc:729] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3cd912c7024ae8ab9aa6b0a6383489, State: Initialized, Role: FOLLOWER
I20260924 10:05:32.148296 30465 raft_consensus.cc:729] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee444dd66464c938c4f3badc3a9f207, State: Initialized, Role: FOLLOWER
I20260924 10:05:32.148934 30465 consensus_queue.cc:260] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.148932 30466 consensus_queue.cc:260] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.150033 30448 heartbeater.cc:503] Master 127.13.162.62:45263 was elected leader, sending a full tablet report...
I20260924 10:05:32.150563 30317 heartbeater.cc:503] Master 127.13.162.62:45263 was elected leader, sending a full tablet report...
I20260924 10:05:32.150767 30465 ts_tablet_manager.cc:1428] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260924 10:05:32.152053 30466 ts_tablet_manager.cc:1428] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 10:05:32.154392 30471 raft_consensus.cc:482] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:05:32.154623 30471 raft_consensus.cc:504] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.156250 30471 leader_election.cc:288] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511), 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683)
I20260924 10:05:32.158442 30470 raft_consensus.cc:482] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:05:32.158690 30470 raft_consensus.cc:504] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.160332 30470 leader_election.cc:288] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905), ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511)
I20260924 10:05:32.161855 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.156545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45511 (local address 127.13.162.3:46587)
0924 10:05:32.156733 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.156752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.156852 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.157351 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.157359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.157386 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.157557 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.157566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.158617 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.158623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.159900 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.159913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.161129 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.161135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.161336 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.161366 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.161445 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.161578 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:05:32.161854 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.156595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46587 (local address 127.13.162.2:45511)
0924 10:05:32.156762 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.156767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.156785 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:05:32.157143 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.157148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.157219 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.157313 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.157674 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.158496 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.160046 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.161000 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.161129 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.161202 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.161480 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.161606 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:05:32.162977 30272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0446de919df5430891441fc691efccee" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" is_pre_election: true
I20260924 10:05:32.163528 30272 raft_consensus.cc:2454] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee444dd66464c938c4f3badc3a9f207 in term 0.
I20260924 10:05:32.164278 30339 leader_election.cc:302] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee444dd66464c938c4f3badc3a9f207, ca3cd912c7024ae8ab9aa6b0a6383489; no voters:
I20260924 10:05:32.165058 30471 raft_consensus.cc:2781] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:05:32.165215 30471 raft_consensus.cc:482] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:05:32.165309 30471 raft_consensus.cc:3037] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:32.168502 30471 raft_consensus.cc:504] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.169025 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.162900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:45511 (local address 127.13.162.1:46035)
0924 10:05:32.163487 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.163503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.163611 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.164059 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.164063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.164085 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.164354 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.164362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.165410 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.165415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.167104 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.167116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.168218 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.168223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.168373 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.168396 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.168474 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.168594 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":21,"thread_start_us":723,"threads_started":1}
I20260924 10:05:32.169652 30272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0446de919df5430891441fc691efccee" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
I20260924 10:05:32.169914 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.160556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:41163 (local address 127.13.162.3:42905)
0924 10:05:32.160718 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.160723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.162829 (+ 2106us) server_negotiation.cc:407] Connection header received
0924 10:05:32.162985 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.162990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.163052 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.163128 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.163867 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.164869 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.167439 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.168660 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.168778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.168847 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.169562 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.169689 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":51}
I20260924 10:05:32.169880 30272 raft_consensus.cc:3037] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:32.170284 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.163353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:46035 (local address 127.13.162.2:45511)
0924 10:05:32.163597 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.163605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.163624 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:05:32.163764 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.163769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.163840 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.163894 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.164487 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.165254 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.167239 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.168024 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.168169 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.168237 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.169771 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.170031 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 10:05:32.170601 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.160536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:42905 (local address 127.13.162.1:41163)
0924 10:05:32.162738 (+ 2202us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.162753 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.162875 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.163276 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.163281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.163307 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.163718 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.163730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.164993 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.164998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.167262 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.167278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.169059 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.169065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.169389 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.169422 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.170065 (+ 643us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.170204 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2086,"mutex_wait_us":21,"thread_start_us":1939,"threads_started":1}
I20260924 10:05:32.170907 30271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0446de919df5430891441fc691efccee" candidate_uuid: "35ae87923a464e94aebd5d5f70266bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" is_pre_election: true
I20260924 10:05:32.171471 30403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0446de919df5430891441fc691efccee" candidate_uuid: "35ae87923a464e94aebd5d5f70266bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee444dd66464c938c4f3badc3a9f207" is_pre_election: true
I20260924 10:05:32.171945 30403 raft_consensus.cc:2379] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35ae87923a464e94aebd5d5f70266bab in current term 1: Already voted for candidate 4ee444dd66464c938c4f3badc3a9f207 in this term.
I20260924 10:05:32.172637 30076 leader_election.cc:302] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35ae87923a464e94aebd5d5f70266bab; no voters: 4ee444dd66464c938c4f3badc3a9f207, ca3cd912c7024ae8ab9aa6b0a6383489
I20260924 10:05:32.173087 30470 raft_consensus.cc:3037] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:05:32.173162 30272 raft_consensus.cc:2454] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee444dd66464c938c4f3badc3a9f207 in term 1.
I20260924 10:05:32.173460 30471 leader_election.cc:288] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 election: Requested vote from peers ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511), 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683)
I20260924 10:05:32.173597 30339 leader_election.cc:302] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee444dd66464c938c4f3badc3a9f207, ca3cd912c7024ae8ab9aa6b0a6383489; no voters:
I20260924 10:05:32.173895 30471 raft_consensus.cc:2781] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:05:32.174468 30471 raft_consensus.cc:686] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee444dd66464c938c4f3badc3a9f207, State: Running, Role: LEADER
I20260924 10:05:32.175356 30471 consensus_queue.cc:237] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } }
I20260924 10:05:32.175797 30470 raft_consensus.cc:2726] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 10:05:32.177785 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.157011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:43949 (local address 127.13.162.1:37683)
0924 10:05:32.157174 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.157180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.157607 (+ 427us) server_negotiation.cc:407] Connection header received
0924 10:05:32.157741 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.157746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.157819 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.157890 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:05:32.160642 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.161882 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.175538 (+ 13656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.176710 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.176868 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.176938 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.177478 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.177579 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:05:32.178085 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.156892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.13.162.3:43949)
0924 10:05:32.157517 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.157530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:05:32.157635 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:05:32.157914 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:05:32.157918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:05:32.157939 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:05:32.158257 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.158270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.162063 (+ 3793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.162067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.175325 (+ 13258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:05:32.175350 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.177129 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.177135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:05:32.177315 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.177348 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:05:32.177444 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:05:32.177580 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":349,"threads_started":1}
I20260924 10:05:32.179095 30139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0446de919df5430891441fc691efccee" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae87923a464e94aebd5d5f70266bab"
I20260924 10:05:32.179126 30140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0446de919df5430891441fc691efccee" candidate_uuid: "4ee444dd66464c938c4f3badc3a9f207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae87923a464e94aebd5d5f70266bab" is_pre_election: true
I20260924 10:05:32.180922 29998 catalog_manager.cc:5495] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3). New cstate: current_term: 1 leader_uuid: "4ee444dd66464c938c4f3badc3a9f207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee444dd66464c938c4f3badc3a9f207" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 42905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 } health_report { overall_health: UNKNOWN } } }
I20260924 10:05:32.181738 30139 raft_consensus.cc:2454] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee444dd66464c938c4f3badc3a9f207 in term 1.
I20260924 10:05:32.199486 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.194704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.13.162.3:42905)
0924 10:05:32.194862 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.194867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.194937 (+ 70us) server_negotiation.cc:407] Connection header received
0924 10:05:32.195104 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.195108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.195207 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.195342 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:05:32.195684 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.196433 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.197676 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.198044 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.198124 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.198550 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:05:32.199039 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:05:32.199146 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.199182 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.199245 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:05:32.207123 30139 raft_consensus.cc:1261] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Refusing update from remote peer 4ee444dd66464c938c4f3badc3a9f207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:05:32.207186 30272 raft_consensus.cc:1261] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Refusing update from remote peer 4ee444dd66464c938c4f3badc3a9f207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:05:32.207878 30476 consensus_queue.cc:1035] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ae87923a464e94aebd5d5f70266bab" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:05:32.208196 30471 consensus_queue.cc:1035] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3cd912c7024ae8ab9aa6b0a6383489" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:05:32.219878 30478 mvcc.cc:205] Tried to move back new op lower bound from 7332840784711483392 to 7332840784595189760. Current Snapshot: MvccSnapshot[applied={T|T < 7332840784711483392}]
I20260924 10:05:32.220064 30479 mvcc.cc:205] Tried to move back new op lower bound from 7332840784711483392 to 7332840784595189760. Current Snapshot: MvccSnapshot[applied={T|T < 7332840784711483392}]
I20260924 10:05:32.221748 30480 mvcc.cc:205] Tried to move back new op lower bound from 7332840784711483392 to 7332840784595189760. Current Snapshot: MvccSnapshot[applied={T|T < 7332840784711483392}]
I20260924 10:05:32.233107 29999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:05:32.233367 29999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:05:32.236526 29999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Sending DeleteTablet for 3 replicas of tablet 0446de919df5430891441fc691efccee
I20260924 10:05:32.237941 30250 tablet_service.cc:1514] Processing DeleteTablet for tablet 0446de919df5430891441fc691efccee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:05:32 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57244
I20260924 10:05:32.237941 30383 tablet_service.cc:1514] Processing DeleteTablet for tablet 0446de919df5430891441fc691efccee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:05:32 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:45724
I20260924 10:05:32.238109 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 30037
I20260924 10:05:32.238142 30120 tablet_service.cc:1514] Processing DeleteTablet for tablet 0446de919df5430891441fc691efccee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:05:32 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:50696
I20260924 10:05:32.250496 30490 tablet_replica.cc:330] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: stopping tablet replica
I20260924 10:05:32.251029 30490 raft_consensus.cc:2229] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:05:32.251673 30490 raft_consensus.cc:2258] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:05:32.252542 30489 tablet_replica.cc:330] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: stopping tablet replica
I20260924 10:05:32.253007 30489 raft_consensus.cc:2229] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:05:32.253695 30489 raft_consensus.cc:2258] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:05:32.254478 30488 tablet_replica.cc:330] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: stopping tablet replica
I20260924 10:05:32.254506 30490 ts_tablet_manager.cc:1905] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:32.254933 30488 raft_consensus.cc:2229] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:05:32.255460 30488 raft_consensus.cc:2258] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:05:32.257587 30489 ts_tablet_manager.cc:1905] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:32.258774 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.238976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.13.162.1:37683)
0924 10:05:32.239158 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.239165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.239605 (+ 440us) server_negotiation.cc:407] Connection header received
0924 10:05:32.239673 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.239679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.239773 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.239859 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:32.241878 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.243178 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.245260 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.245988 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.246103 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.250108 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:32.250135 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:32.250160 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:32.250206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:32.254180 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.255351 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.255360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.255363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.255536 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.256099 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.256104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.256107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.256593 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:32.256909 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.258119 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.258393 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:05:32.259657 30180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:32.260629 30488 ts_tablet_manager.cc:1905] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:32.265560 30488 ts_tablet_manager.cc:1918] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:05:32.265751 30488 log.cc:1196] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/0446de919df5430891441fc691efccee
I20260924 10:05:32.266276 30488 ts_tablet_manager.cc:1939] T 0446de919df5430891441fc691efccee P ca3cd912c7024ae8ab9aa6b0a6383489: Deleting consensus metadata
I20260924 10:05:32.268193 29987 catalog_manager.cc:4841] TS ca3cd912c7024ae8ab9aa6b0a6383489 (127.13.162.2:45511): tablet 0446de919df5430891441fc691efccee (table test-table [id=70ef68b9b48f47639f44a060e403c86d]) successfully deleted
I20260924 10:05:32.272257 30489 ts_tablet_manager.cc:1918] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:05:32.272544 30489 log.cc:1196] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/0446de919df5430891441fc691efccee
I20260924 10:05:32.273099 30489 ts_tablet_manager.cc:1939] T 0446de919df5430891441fc691efccee P 4ee444dd66464c938c4f3badc3a9f207: Deleting consensus metadata
I20260924 10:05:32.275076 29987 catalog_manager.cc:4841] TS 4ee444dd66464c938c4f3badc3a9f207 (127.13.162.3:42905): tablet 0446de919df5430891441fc691efccee (table test-table [id=70ef68b9b48f47639f44a060e403c86d]) successfully deleted
I20260924 10:05:32.358415 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 30037
I20260924 10:05:32.374360 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 30189
W20260924 10:05:32.374639 29987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:37683 (error 108)
I20260924 10:05:32.377517 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.377060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50706)
0924 10:05:32.377238 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.377351 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:05:32.378626 29987 catalog_manager.cc:4584] TS 35ae87923a464e94aebd5d5f70266bab (127.13.162.1:37683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0446de919df5430891441fc691efccee: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
I20260924 10:05:32.388685 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.375477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.13.162.2:45511)
0924 10:05:32.375667 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.375673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.375693 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:05:32.375758 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.375763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.375858 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.375967 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:32.377526 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.378358 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.379874 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.380288 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.380398 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.383018 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:32.383032 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:32.383046 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:32.383083 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:32.385925 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.386710 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.386720 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.386724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.386855 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.387216 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.387222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.387226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.387585 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:32.387868 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.388176 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.388385 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:05:32.389271 30312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:32.432313 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.431767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50712)
0924 10:05:32.432004 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.432127 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 10:05:32.433485 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.433087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50726)
0924 10:05:32.433246 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.433348 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:05:32.481452 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 30189
I20260924 10:05:32.494257 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.493741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50736)
0924 10:05:32.493973 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.494076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 10:05:32.495397 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.495019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50744)
0924 10:05:32.495192 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.495278 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:05:32.501781 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 30320
I20260924 10:05:32.521567 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.506673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.13.162.3:42905)
0924 10:05:32.506899 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:05:32.506906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:05:32.506931 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:05:32.507026 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:05:32.507032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:05:32.507126 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:05:32.507239 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:05:32.508497 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.509591 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.510908 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:05:32.511569 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:05:32.511714 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:05:32.514168 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:05:32.514184 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:05:32.514205 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:05:32.514250 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:05:32.518397 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.519240 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.519250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.519253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.519426 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:05:32.519895 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:05:32.519899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:05:32.519901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:05:32.520343 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:05:32.520649 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 10:05:32.520990 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 10:05:32.521229 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:05:32.522243 30443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:32.576128 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.575564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50748)
0924 10:05:32.575853 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.575955 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 10:05:32.577184 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.576809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50752)
0924 10:05:32.576988 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.577061 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 10:05:32.643709 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 30320
I20260924 10:05:32.660065 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 29967
I20260924 10:05:32.661154 30030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:05:32.773391 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:32.772874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:37683 (local address 127.0.0.1:50766)
0924 10:05:32.773092 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:32.773203 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:05:32.773404 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 29967
I20260924 10:05:52.791381 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45263
--webserver_interface=127.13.162.62
--webserver_port=40591
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:05:53.122577 30496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:53.122892 30496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:53.123031 30496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:53.130546 30496 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:05:53.130676 30496 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:05:53.130739 30496 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:05:53.130812 30496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:05:53.130887 30496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:53.130985 30496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:53.131045 30496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:05:53.131119 30496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:05:53.141614 30496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=40591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:53.143465 30496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:53.145201 30496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:53.156926 30501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:53.156989 30502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:53.156953 30504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:53.157646 30496 server_base.cc:1034] running on GCE node
I20260924 10:05:53.158469 30496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:53.159895 30496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:53.161119 30496 hybrid_clock.cc:648] HybridClock initialized: now 1790244353161050 us; error 46 us; skew 500 ppm
I20260924 10:05:53.161448 30496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:53 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244353, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:53.168172 30496 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:05:53.169657 30496 webserver.cc:460] Webserver started at http://127.13.162.62:40591/ using document root <none> and password file <none>
I20260924 10:05:53.170413 30496 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:53.170519 30496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:53.175926 30496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:05:53.178699 30511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:53.180092 30496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:05:53.180260 30496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "df0f317e5544467bb59d733201ef0881"
format_stamp: "Formatted at 2026-09-24 10:05:07 on dist-test-slave-jvsj"
server_key: "396ca37e66ba6c54169cc2ed0a221c88"
server_key_iv: "59ffd1422a802d8c511b3f5bc5746b58"
server_key_version: "encryptionkey@0"
I20260924 10:05:53.181305 30496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:53.211664 30496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:53.215833 30496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:53.216128 30496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:53.234241 30496 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:45263
I20260924 10:05:53.234268 30562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:45263 every 8 connection(s)
I20260924 10:05:53.235813 30496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:05:53.242738 30563 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:05:53.244320 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30496
I20260924 10:05:53.245318 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:37683
--local_ip_for_outbound_sockets=127.13.162.1
--tserver_master_addrs=127.13.162.62:45263
--webserver_port=39473
--webserver_interface=127.13.162.1
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:05:53.246975 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap starting.
I20260924 10:05:53.268750 30563 log.cc:824] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Log is configured to *not* fsync() on all Append() calls
I20260924 10:05:53.285356 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:05:53.285986 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Bootstrap complete.
I20260924 10:05:53.296145 30563 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:53.297194 30563 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0f317e5544467bb59d733201ef0881, State: Initialized, Role: FOLLOWER
I20260924 10:05:53.297904 30563 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:53.298203 30563 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:05:53.298360 30563 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:05:53.298545 30563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:05:53.303577 30563 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:53.304185 30563 leader_election.cc:302] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0f317e5544467bb59d733201ef0881; no voters:
I20260924 10:05:53.304826 30563 leader_election.cc:288] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:05:53.305002 30567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:05:53.306252 30567 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [term 3 LEADER]: Becoming Leader. State: Replica: df0f317e5544467bb59d733201ef0881, State: Running, Role: LEADER
I20260924 10:05:53.306957 30567 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } }
I20260924 10:05:53.307164 30563 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:05:53.309634 30569 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0f317e5544467bb59d733201ef0881. Latest consensus state: current_term: 3 leader_uuid: "df0f317e5544467bb59d733201ef0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } } }
I20260924 10:05:53.310429 30569 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: This master's current role is: LEADER
I20260924 10:05:53.310086 30568 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "df0f317e5544467bb59d733201ef0881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0f317e5544467bb59d733201ef0881" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45263 } } }
I20260924 10:05:53.310971 30568 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881 [sys.catalog]: This master's current role is: LEADER
I20260924 10:05:53.320910 30574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:05:53.331045 30574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:05:53.333170 30574 catalog_manager.cc:1259] Loaded cluster ID: 69ddfb780cb8423481ba681d1fb2be43
I20260924 10:05:53.333932 30574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:05:53.338480 30574 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:05:53.340446 30574 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Loaded TSK: 0
I20260924 10:05:53.340735 30574 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0f317e5544467bb59d733201ef0881: Loaded TSK: 1
F20260924 10:05:53.342160 30574 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe655b4c9ad google::LogMessage::Fail() at ??:0
@ 0x7fe655b4f88a google::LogMessage::SendToLog() at ??:0
I20260924 10:05:53.340572 30584 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fe655b4c4de google::LogMessage::Flush() at ??:0
@ 0x7fe655b4da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe6598b07d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe65989f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe66b57f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe66b57e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe66b60a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe66b60a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe66b6545e9 std::function<>::operator()() at ??:0
@ 0x7fe66b582cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe66b580c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe66b608610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe66b608439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe658798345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe65879b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe65879b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe658776eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe65ba506db start_thread at ??:0
@ 0x7fe6549e661f clone at ??:0
W20260924 10:05:53.626168 30565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:53.626551 30565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:53.626708 30565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:53.634718 30565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:53.634850 30565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:53.635002 30565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:05:53.645200 30565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:37683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=39473
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:53.646587 30565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:53.648209 30565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:53.661706 30610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:53.661864 30613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:53.661890 30611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:53.663273 30565 server_base.cc:1034] running on GCE node
I20260924 10:05:53.663992 30565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:53.665014 30565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:53.666226 30565 hybrid_clock.cc:648] HybridClock initialized: now 1790244353666170 us; error 58 us; skew 500 ppm
I20260924 10:05:53.666579 30565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:53 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244353, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:53.670054 30565 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:05:53.671726 30565 webserver.cc:460] Webserver started at http://127.13.162.1:39473/ using document root <none> and password file <none>
I20260924 10:05:53.672449 30565 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:53.672559 30565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:53.679056 30565 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:05:53.682188 30620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:53.683781 30565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:05:53.683935 30565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "35ae87923a464e94aebd5d5f70266bab"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "5322439ba72fe68c0f7c7a61eb3b8d22"
server_key_iv: "868d916c78d40ec93ceaa9bf1a775ce8"
server_key_version: "encryptionkey@0"
I20260924 10:05:53.684875 30565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:53.700453 30565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:53.705060 30565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:53.705330 30565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:53.706434 30565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:53.708372 30627 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 10:05:53.714082 30627 ts_tablet_manager.cc:1852] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 10:05:53.715127 30627 ts_tablet_manager.cc:1905] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:05:53.722740 30627 ts_tablet_manager.cc:1918] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:05:53.722929 30627 log.cc:1196] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/0446de919df5430891441fc691efccee
I20260924 10:05:53.723328 30627 ts_tablet_manager.cc:1939] T 0446de919df5430891441fc691efccee P 35ae87923a464e94aebd5d5f70266bab: Deleting consensus metadata
I20260924 10:05:53.723660 30565 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 10:05:53.723778 30565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260924 10:05:53.723948 30565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:53.724020 30565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:53.755431 30565 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:37683
I20260924 10:05:53.755477 30733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:37683 every 8 connection(s)
I20260924 10:05:53.756816 30565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:05:53.761780 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:53.759803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:47417)
0924 10:05:53.760629 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:53.760883 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":309,"threads_started":1}
I20260924 10:05:53.762238 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30565
W20260924 10:05:53.762441 30734 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:45263 (0 consecutive failures): Network error: Failed to ping master at 127.13.162.62:45263: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
I20260924 10:05:53.763039 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:45511
--local_ip_for_outbound_sockets=127.13.162.2
--tserver_master_addrs=127.13.162.62:45263
--webserver_port=33371
--webserver_interface=127.13.162.2
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:05:53.763365 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:53.762991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:44185)
0924 10:05:53.763148 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:53.763251 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:05:53.764343 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:53.763876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:44625)
0924 10:05:53.764069 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:53.764149 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 10:05:53.764587 30734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:05:54.083860 30737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:54.084156 30737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:54.084254 30737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:54.091645 30737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:54.091770 30737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:54.091892 30737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:05:54.101815 30737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:45511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=33371
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:54.103224 30737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:54.104732 30737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:54.117602 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:54.117662 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:54.117640 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:54.118345 30737 server_base.cc:1034] running on GCE node
I20260924 10:05:54.118923 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:54.119848 30737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:54.121050 30737 hybrid_clock.cc:648] HybridClock initialized: now 1790244354120989 us; error 51 us; skew 500 ppm
I20260924 10:05:54.121336 30737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:54 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244354, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:54.124388 30737 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:05:54.125800 30737 webserver.cc:460] Webserver started at http://127.13.162.2:33371/ using document root <none> and password file <none>
I20260924 10:05:54.126509 30737 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:54.126618 30737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:54.131944 30737 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:05:54.134752 30752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:54.136210 30737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:05:54.136356 30737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "ca3cd912c7024ae8ab9aa6b0a6383489"
format_stamp: "Formatted at 2026-09-24 10:05:08 on dist-test-slave-jvsj"
server_key: "6764d3c06fecbee950ff455ed0fbef93"
server_key_iv: "1cd3f48707c1f6308cfc9ec762332101"
server_key_version: "encryptionkey@0"
I20260924 10:05:54.137326 30737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:54.154682 30737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:54.158932 30737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:54.159154 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:54.160097 30737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:54.161368 30737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:54.161458 30737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:54.161535 30737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:54.161592 30737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:54.192039 30737 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:45511
I20260924 10:05:54.192113 30864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:45511 every 8 connection(s)
I20260924 10:05:54.193485 30737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:05:54.195576 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30737
I20260924 10:05:54.196210 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:42905
--local_ip_for_outbound_sockets=127.13.162.3
--tserver_master_addrs=127.13.162.62:45263
--webserver_port=37075
--webserver_interface=127.13.162.3
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:05:54.198606 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.196526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:45177)
0924 10:05:54.197342 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.197628 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":309,"threads_started":1}
W20260924 10:05:54.199208 30865 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:45263 (0 consecutive failures): Network error: Failed to ping master at 127.13.162.62:45263: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
I20260924 10:05:54.200083 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.199772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:58423)
0924 10:05:54.199912 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.199981 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:05:54.200884 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.200599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:55197)
0924 10:05:54.200710 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.200781 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:05:54.201136 30865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:05:54.526141 30868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:05:54.526500 30868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:05:54.526610 30868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:05:54.533890 30868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:05:54.534017 30868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:05:54.534129 30868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:05:54.544454 30868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:42905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=37075
--tserver_master_addrs=127.13.162.62:45263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:05:54.545745 30868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:05:54.547112 30868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:05:54.560021 30873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:54.560721 30868 server_base.cc:1034] running on GCE node
W20260924 10:05:54.560758 30876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:05:54.560844 30874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:05:54.561523 30868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:05:54.562531 30868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:05:54.563740 30868 hybrid_clock.cc:648] HybridClock initialized: now 1790244354563677 us; error 53 us; skew 500 ppm
I20260924 10:05:54.564064 30868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:05:54 dist-test-slave-jvsj krb5kdc[29414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244354, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:05:54.567427 30868 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:05:54.568899 30868 webserver.cc:460] Webserver started at http://127.13.162.3:37075/ using document root <none> and password file <none>
I20260924 10:05:54.569594 30868 fs_manager.cc:359] Metadata directory not provided
I20260924 10:05:54.569713 30868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:05:54.575160 30868 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:05:54.578109 30883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:54.579578 30868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:05:54.579731 30868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "4ee444dd66464c938c4f3badc3a9f207"
format_stamp: "Formatted at 2026-09-24 10:05:09 on dist-test-slave-jvsj"
server_key: "3b6a39fe7b97de9773422d4df63793a1"
server_key_iv: "db86a13d7e7516f1243a67e9dc51e4f1"
server_key_version: "encryptionkey@0"
I20260924 10:05:54.580719 30868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:05:54.595721 30868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:05:54.599876 30868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:05:54.600137 30868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:05:54.601126 30868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:05:54.602613 30868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:05:54.602708 30868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:54.602811 30868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:05:54.602901 30868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:05:54.633599 30868 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:42905
I20260924 10:05:54.633675 30995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:42905 every 8 connection(s)
I20260924 10:05:54.635169 30868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:05:54.639189 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 30868
I20260924 10:05:54.640146 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.638089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:38045)
0924 10:05:54.638804 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.639049 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":288,"threads_started":1}
W20260924 10:05:54.641067 30996 heartbeater.cc:644] Failed to heartbeat to 127.13.162.62:45263 (0 consecutive failures): Network error: Failed to ping master at 127.13.162.62:45263: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
I20260924 10:05:54.641911 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.641578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:40951)
0924 10:05:54.641734 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.641800 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:05:54.642724 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.642428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:40799)
0924 10:05:54.642553 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.642623 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:05:54.642937 30996 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:05:54.765863 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:54.765107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:58485)
0924 10:05:54.765531 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:54.765628 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":203,"threads_started":1}
I20260924 10:05:55.202601 31001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:55.201666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:45939)
0924 10:05:55.202143 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:55.202259 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260924 10:05:55.644371 31002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:55.643510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:55519)
0924 10:05:55.644031 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:55.644138 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":268,"threads_started":1}
I20260924 10:05:55.767681 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:55.766921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:52833)
0924 10:05:55.767430 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:55.767523 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":252,"threads_started":1}
I20260924 10:05:56.204245 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:56.203512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:59691)
0924 10:05:56.203982 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:56.204074 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260924 10:05:56.645888 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:56.645205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:42897)
0924 10:05:56.645647 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:56.645724 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260924 10:05:56.769282 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:56.768560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:40991)
0924 10:05:56.769018 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:56.769106 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260924 10:05:57.205750 31007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:57.205061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:47797)
0924 10:05:57.205505 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:57.205584 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260924 10:05:57.647382 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:57.646695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:48589)
0924 10:05:57.647125 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:57.647207 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260924 10:05:57.770975 31009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:57.770131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:58693)
0924 10:05:57.770717 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:57.770807 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":301,"threads_started":1}
I20260924 10:05:58.207532 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:58.206730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:41087)
0924 10:05:58.207225 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:58.207326 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260924 10:05:58.649113 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:58.648331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:38603)
0924 10:05:58.648816 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:58.648906 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":243,"threads_started":1}
I20260924 10:05:58.772733 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:58.771954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:47219)
0924 10:05:58.772469 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:58.772561 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
I20260924 10:05:59.209110 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:59.208410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:56465)
0924 10:05:59.208861 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:59.208942 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":223,"threads_started":1}
I20260924 10:05:59.650827 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:59.650080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:53369)
0924 10:05:59.650570 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:59.650659 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260924 10:05:59.774504 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:05:59.773649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:58775)
0924 10:05:59.774149 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 10:05:59.774254 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
I20260924 10:06:00.210791 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:00.210019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:35383)
0924 10:06:00.210525 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:00.210624 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":231,"threads_started":1}
I20260924 10:06:00.652637 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:00.651865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:45591)
0924 10:06:00.652343 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:00.652441 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260924 10:06:00.776230 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:00.775415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:35673)
0924 10:06:00.775926 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:00.776026 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":233,"threads_started":1}
I20260924 10:06:01.212671 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:01.211794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:39863)
0924 10:06:01.212327 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:01.212427 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":235,"threads_started":1}
I20260924 10:06:01.654359 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:01.653475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:42123)
0924 10:06:01.654007 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:01.654118 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":270,"threads_started":1}
I20260924 10:06:01.777874 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:01.777198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:37357)
0924 10:06:01.777619 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:01.777707 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260924 10:06:02.214578 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:02.213795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:46799)
0924 10:06:02.214283 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:02.214402 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260924 10:06:02.656100 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:02.655306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:46627)
0924 10:06:02.655781 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:02.655865 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260924 10:06:02.779500 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:02.778775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:56621)
0924 10:06:02.779229 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:02.779332 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":205,"threads_started":1}
I20260924 10:06:03.216189 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:03.215431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:44257)
0924 10:06:03.215909 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:03.216013 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":231,"threads_started":1}
I20260924 10:06:03.657755 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:03.656983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:57073)
0924 10:06:03.657488 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:03.657571 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":239,"threads_started":1}
I20260924 10:06:03.781101 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:03.780392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:43253)
0924 10:06:03.780844 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:03.780931 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260924 10:06:04.217758 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:04.217032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:41497)
0924 10:06:04.217509 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:04.217597 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260924 10:06:04.659475 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:04.658766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:51343)
0924 10:06:04.659226 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:04.659313 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260924 10:06:04.782817 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:04.782052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:36971)
0924 10:06:04.782535 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:04.782623 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":275,"threads_started":1}
I20260924 10:06:05.219610 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:05.218796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:48477)
0924 10:06:05.219253 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:05.219351 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":217,"threads_started":1}
I20260924 10:06:05.661197 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:05.660459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:33569)
0924 10:06:05.660916 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:05.661014 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260924 10:06:05.784485 31033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:05.783719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:52991)
0924 10:06:05.784225 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:05.784317 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260924 10:06:06.221302 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:06.220589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:48473)
0924 10:06:06.221056 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:06.221147 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260924 10:06:06.662904 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:06.662198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:52833)
0924 10:06:06.662654 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:06.662748 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
I20260924 10:06:06.786346 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:06.785402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:37995)
0924 10:06:06.785950 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:06.786060 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":302,"threads_started":1}
I20260924 10:06:07.223183 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:07.222261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:50673)
0924 10:06:07.222848 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:07.222977 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
I20260924 10:06:07.664613 31038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:07.663849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:45765)
0924 10:06:07.664331 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:07.664428 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":236,"threads_started":1}
I20260924 10:06:07.787997 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:07.787221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:57649)
0924 10:06:07.787692 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:07.787811 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":227,"threads_started":1}
I20260924 10:06:08.224838 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:08.224083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:47339)
0924 10:06:08.224524 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:08.224610 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":216,"threads_started":1}
I20260924 10:06:08.666244 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:08.665517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:57511)
0924 10:06:08.665977 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:08.666061 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":247,"threads_started":1}
I20260924 10:06:08.789888 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:08.789108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.1:59093)
0924 10:06:08.789595 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:08.789695 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260924 10:06:09.226794 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:09.225965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.2:33207)
0924 10:06:09.226472 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:09.226573 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":237,"threads_started":1}
I20260924 10:06:09.642444 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 30565
I20260924 10:06:09.660485 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:09.643502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.13.162.1:37683)
0924 10:06:09.644595 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:09.644615 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:09.644661 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:06:09.646258 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:09.646270 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:09.646759 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:09.647102 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:09.648565 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:09.649844 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:09.651199 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:09.651891 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:09.652029 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:09.654784 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:09.654793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:09.654809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:09.654852 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:09.657634 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:09.658252 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:09.658258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:09.658265 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:09.658467 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:09.658790 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:09.658794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:09.658795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:09.659303 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:09.659524 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:09.659870 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 10:06:09.660193 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":759,"threads_started":1}
I20260924 10:06:09.661864 30729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:09.671139 31046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:09.667318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:45263 (local address 127.13.162.3:38235)
0924 10:06:09.670773 (+ 3455us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:09.670896 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.62:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3296,"thread_start_us":3168,"threads_started":1}
I20260924 10:06:09.745517 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 30565
I20260924 10:06:09.759648 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 30737
I20260924 10:06:09.776798 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:09.760599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.13.162.2:45511)
0924 10:06:09.761157 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:09.761172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:09.761209 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:06:09.762801 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:09.762812 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:09.763249 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:09.763584 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:09.765016 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:09.766241 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:09.767568 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:09.768262 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:09.768446 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:09.770919 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:09.770927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:09.770942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:09.770984 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:09.773653 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:09.774287 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:09.774294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:09.774300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:09.774518 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:09.774873 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:09.774878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:09.774881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:09.775381 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:09.775661 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:09.776097 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:06:09.776504 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260924 10:06:09.778147 30860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:09.859627 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 30737
I20260924 10:06:09.874531 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 30868
I20260924 10:06:09.896843 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:09.875572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.13.162.3:42905)
0924 10:06:09.876173 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:09.876192 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:09.876238 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:06:09.877836 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:09.877845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:09.878292 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:09.878692 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:09.880027 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:09.881256 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:09.882565 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:09.883268 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:09.883442 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:09.890820 (+ 7378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:09.890828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:09.890846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:09.890900 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:09.893787 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:09.894525 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:09.894532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:09.894539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:09.894714 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:09.895018 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:09.895021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:09.895023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:09.895562 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:09.895785 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:09.896099 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 10:06:09.896548 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":278,"threads_started":1}
I20260924 10:06:09.898196 30991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:09.980544 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 30868
2026-09-24T10:06:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64064 ms)
[----------] 30 tests from SecurityITest (283979 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:10 dist-test-slave-jvsj krb5kdc[31053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:10 dist-test-slave-jvsj krb5kdc[31053](info): set up 2 sockets
Sep 24 10:06:10 dist-test-slave-jvsj krb5kdc[31053](info): commencing operation
krb5kdc: starting...
W20260924 10:06:10.994372 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:11 dist-test-slave-jvsj krb5kdc[31053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:11.226439 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37195
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:11.548310 31066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:11.548635 31066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:11.548774 31066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:11.556128 31066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:11.556281 31066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:11.556360 31066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:11.556437 31066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:11.556496 31066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:11.566346 31066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:11.567631 31066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:11.568904 31066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:11.580026 31071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:11.580054 31072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:11.580083 31074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:11.581003 31066 server_base.cc:1034] running on GCE node
I20260924 10:06:11.581765 31066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:11.583245 31066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:11.584450 31066 hybrid_clock.cc:648] HybridClock initialized: now 1790244371584403 us; error 39 us; skew 500 ppm
I20260924 10:06:11.584789 31066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:11 dist-test-slave-jvsj krb5kdc[31053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244371, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:11.587976 31066 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:06:11.589460 31066 webserver.cc:460] Webserver started at http://127.13.162.62:39349/ using document root <none> and password file <none>
I20260924 10:06:11.590255 31066 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:11.590400 31066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:11.590682 31066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:11.593698 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec"
format_stamp: "Formatted at 2026-09-24 10:06:11 on dist-test-slave-jvsj"
server_key: "27de955dedaf5fed43836e8c4ac831db"
server_key_iv: "e57ac62884a421b8752cbb6dcc45e12b"
server_key_version: "encryptionkey@0"
I20260924 10:06:11.594462 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec"
format_stamp: "Formatted at 2026-09-24 10:06:11 on dist-test-slave-jvsj"
server_key: "27de955dedaf5fed43836e8c4ac831db"
server_key_iv: "e57ac62884a421b8752cbb6dcc45e12b"
server_key_version: "encryptionkey@0"
I20260924 10:06:11.599764 31066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:06:11.603148 31081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:11.604487 31066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:11.604637 31066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec"
format_stamp: "Formatted at 2026-09-24 10:06:11 on dist-test-slave-jvsj"
server_key: "27de955dedaf5fed43836e8c4ac831db"
server_key_iv: "e57ac62884a421b8752cbb6dcc45e12b"
server_key_version: "encryptionkey@0"
I20260924 10:06:11.604809 31066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:11.622303 31066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:11.626382 31066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:11.626643 31066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:11.643174 31066 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:37195
I20260924 10:06:11.643185 31132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:37195 every 8 connection(s)
I20260924 10:06:11.644580 31066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:11.648077 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31066
I20260924 10:06:11.648252 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:11.648579 31133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:11.648892 13960 external_mini_cluster.cc:1428] Setting key 0df4bf77c78575c769a944a660e21bf1
I20260924 10:06:11.661046 31133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec: Bootstrap starting.
Sep 24 10:06:11 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:11.664269 31133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:11.665369 31133 log.cc:824] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:11.668143 31133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec: No bootstrap required, opened a new log
I20260924 10:06:11.669927 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:11.650806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.13.162.62:37195)
0924 10:06:11.651602 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:11.651614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:11.651653 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:06:11.653372 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:11.653394 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:11.653788 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:11.654157 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:11.655612 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:11.656868 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:11.658120 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:11.658854 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:11.659038 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:11.662996 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:11.663005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:11.663019 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:11.663057 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:11.665992 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:11.666732 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:11.666739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:11.666751 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:11.666923 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:11.667295 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:11.667302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:11.667304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:11.667820 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:11.668100 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:11.668536 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:06:11.668959 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":345,"threads_started":1}
I20260924 10:06:11.676306 31133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37195 } }
I20260924 10:06:11.676764 31133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:11.676872 31133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cbf6aea72415eb6aeafdb1d2e74ec, State: Initialized, Role: FOLLOWER
I20260924 10:06:11.677491 31133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37195 } }
I20260924 10:06:11.677701 31133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:11.677790 31133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:11.677919 31133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:11.679999 31133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37195 } }
I20260924 10:06:11.680430 31133 leader_election.cc:302] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f2cbf6aea72415eb6aeafdb1d2e74ec; no voters:
I20260924 10:06:11.680960 31133 leader_election.cc:288] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:11.681094 31138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:11.681679 31138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cbf6aea72415eb6aeafdb1d2e74ec, State: Running, Role: LEADER
I20260924 10:06:11.682214 31138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37195 } }
I20260924 10:06:11.682866 31133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:11.684911 31140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f2cbf6aea72415eb6aeafdb1d2e74ec. Latest consensus state: current_term: 1 leader_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37195 } } }
I20260924 10:06:11.684902 31139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2cbf6aea72415eb6aeafdb1d2e74ec" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37195 } } }
I20260924 10:06:11.685333 31140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:11.685460 31139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:11.687656 31143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:11.692118 31143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:11.700357 31143 catalog_manager.cc:1347] Generated new cluster ID: 62910d89407b437da4b11bd5cbf7e528
I20260924 10:06:11.700475 31143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:11.714522 31143 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:11.715912 31143 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:11.740958 31143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec: Generated new TSK 0
I20260924 10:06:11.741852 31143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:06:11.797927 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37195
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:12.114943 31159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:12.115289 31159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:12.115420 31159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:12.123178 31159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:12.123281 31159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:12.123427 31159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:06:12.133459 31159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:12.134830 31159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:12.136106 31159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:12.149091 31164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:12.149149 31167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:12.149130 31165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:12.149755 31159 server_base.cc:1034] running on GCE node
I20260924 10:06:12.150425 31159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:12.151358 31159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:12.152570 31159 hybrid_clock.cc:648] HybridClock initialized: now 1790244372152502 us; error 56 us; skew 500 ppm
I20260924 10:06:12.152921 31159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:12 dist-test-slave-jvsj krb5kdc[31053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244372, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:12.156073 31159 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:06:12.157559 31159 webserver.cc:460] Webserver started at http://127.13.162.1:44123/ using document root <none> and password file <none>
I20260924 10:06:12.158252 31159 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:12.158380 31159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:12.158689 31159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:12.161453 31159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "21abc50311284c039209ae7b3df50b73"
format_stamp: "Formatted at 2026-09-24 10:06:12 on dist-test-slave-jvsj"
server_key: "3480e1db7d912e21034f95b60f6bd10d"
server_key_iv: "74ad1879b4b97dcb391fcc2169c872fd"
server_key_version: "encryptionkey@0"
I20260924 10:06:12.162128 31159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "21abc50311284c039209ae7b3df50b73"
format_stamp: "Formatted at 2026-09-24 10:06:12 on dist-test-slave-jvsj"
server_key: "3480e1db7d912e21034f95b60f6bd10d"
server_key_iv: "74ad1879b4b97dcb391fcc2169c872fd"
server_key_version: "encryptionkey@0"
I20260924 10:06:12.167411 31159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:06:12.170831 31174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:12.172252 31159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:06:12.172398 31159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "21abc50311284c039209ae7b3df50b73"
format_stamp: "Formatted at 2026-09-24 10:06:12 on dist-test-slave-jvsj"
server_key: "3480e1db7d912e21034f95b60f6bd10d"
server_key_iv: "74ad1879b4b97dcb391fcc2169c872fd"
server_key_version: "encryptionkey@0"
I20260924 10:06:12.172574 31159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:12.197335 31159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:12.201388 31159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:12.201646 31159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:12.202602 31159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:12.203864 31159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:12.203935 31159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:12.204051 31159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:12.204100 31159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:12.235816 31159 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:46259
I20260924 10:06:12.235878 31286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:46259 every 8 connection(s)
I20260924 10:06:12.237094 31159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:06:12.239539 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31159
I20260924 10:06:12.239755 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:06:12.240423 13960 external_mini_cluster.cc:1428] Setting key 1eaacbf157bb040b2965bf9c2541fb27
Sep 24 10:06:12 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244372, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:12.258225 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:12.239628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:39281 (local address 127.13.162.62:37195)
0924 10:06:12.240075 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:12.240081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:12.241449 (+ 1368us) server_negotiation.cc:407] Connection header received
0924 10:06:12.243364 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:12.243368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:12.243434 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:12.243521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:12.245405 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.246088 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.248113 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.248513 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.248600 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:12.252416 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:12.252422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:12.252426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:12.252464 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:12.255352 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:12.256010 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:12.256016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:12.256019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:12.256151 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:12.256562 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:12.256566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:12.256570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:12.256869 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:12.256996 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:12.257646 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 10:06:12.257877 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":205,"threads_started":1}
I20260924 10:06:12.259021 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:12.240215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37195 (local address 127.13.162.1:39281)
0924 10:06:12.241059 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:12.241108 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:12.243083 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:12.243767 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:12.243779 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:12.244490 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:12.245242 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:12.245266 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.246222 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.246227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:12.247969 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:12.247983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.248711 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.248717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:12.248967 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:12.249642 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:12.249668 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:12.252186 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:12.255469 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:12.255476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:12.255489 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:12.255895 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:12.256250 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:12.256253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:12.256255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:12.256463 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:12.256965 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:12.256970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:12.257268 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 10:06:12.257515 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 10:06:12.257897 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":351,"threads_started":1}
I20260924 10:06:12.260473 31287 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37195
I20260924 10:06:12.260803 31287 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:12.261623 31287 heartbeater.cc:511] Master 127.13.162.62:37195 requested a full tablet report, sending...
I20260924 10:06:12.263873 31098 ts_manager.cc:194] Registered new tserver with Master: 21abc50311284c039209ae7b3df50b73 (127.13.162.1:46259)
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:06:12.265383 31098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:39281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:06:12.292891 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37195
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:12.605777 31293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:12.606036 31293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:12.606129 31293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:12.613214 31293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:12.613317 31293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:12.613441 31293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:06:12.622997 31293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:12.624198 31293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:12.625464 31293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:12.637990 31298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:12.638051 31299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:12.637990 31301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:12.639213 31293 server_base.cc:1034] running on GCE node
I20260924 10:06:12.639823 31293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:12.640744 31293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:12.641942 31293 hybrid_clock.cc:648] HybridClock initialized: now 1790244372641896 us; error 36 us; skew 500 ppm
I20260924 10:06:12.642247 31293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:12 dist-test-slave-jvsj krb5kdc[31053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244372, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:12.645874 31293 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:06:12.647492 31293 webserver.cc:460] Webserver started at http://127.13.162.2:45261/ using document root <none> and password file <none>
I20260924 10:06:12.648289 31293 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:12.648425 31293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:12.648756 31293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:12.652478 31293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "baeb1a7730ea4f04834517c49178708b"
format_stamp: "Formatted at 2026-09-24 10:06:12 on dist-test-slave-jvsj"
server_key: "ec68bdb6c664ae9b37554187293f4708"
server_key_iv: "96fec268304ffedb508695626580071e"
server_key_version: "encryptionkey@0"
I20260924 10:06:12.653365 31293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "baeb1a7730ea4f04834517c49178708b"
format_stamp: "Formatted at 2026-09-24 10:06:12 on dist-test-slave-jvsj"
server_key: "ec68bdb6c664ae9b37554187293f4708"
server_key_iv: "96fec268304ffedb508695626580071e"
server_key_version: "encryptionkey@0"
I20260924 10:06:12.659590 31293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 10:06:12.663630 31308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:12.664918 31293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:12.665053 31293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "baeb1a7730ea4f04834517c49178708b"
format_stamp: "Formatted at 2026-09-24 10:06:12 on dist-test-slave-jvsj"
server_key: "ec68bdb6c664ae9b37554187293f4708"
server_key_iv: "96fec268304ffedb508695626580071e"
server_key_version: "encryptionkey@0"
I20260924 10:06:12.665235 31293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:12.682806 31293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:12.686604 31293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:12.686856 31293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:12.687809 31293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:12.689163 31293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:12.689234 31293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:12.689355 31293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:12.689402 31293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:12.719374 31293 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:42991
I20260924 10:06:12.719415 31420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:42991 every 8 connection(s)
I20260924 10:06:12.720697 31293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:06:12.723770 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31293
I20260924 10:06:12.723940 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:06:12.724334 13960 external_mini_cluster.cc:1428] Setting key c642979cec4e84b11d7f6bad03156d22
Sep 24 10:06:12 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244372, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:12.740862 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:12.723236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:45147 (local address 127.13.162.62:37195)
0924 10:06:12.723424 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:12.723430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:12.724706 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 10:06:12.726743 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:12.726748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:12.726842 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:12.727029 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:12.728563 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.729549 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.731132 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.731730 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.731849 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:12.735251 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:12.735257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:12.735260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:12.735294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:12.737752 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:12.738602 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:12.738606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:12.738609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:12.738763 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:12.739186 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:12.739189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:12.739191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:12.739490 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:12.739597 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:12.740442 (+ 845us) server_negotiation.cc:299] Negotiation successful
0924 10:06:12.740614 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:06:12.742166 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:12.723726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37195 (local address 127.13.162.2:45147)
0924 10:06:12.724528 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:12.724576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:12.726465 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:12.727230 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:12.727240 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:12.727793 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:12.728381 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:12.728400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.729690 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.729694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:12.730990 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:12.731013 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:12.731924 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:12.731929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:12.732133 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:12.732758 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:12.732785 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:12.735034 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:12.737889 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:12.737899 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:12.737917 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:12.738496 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:12.738858 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:12.738863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:12.738866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:12.739095 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:12.739598 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:12.739605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:12.740039 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 10:06:12.740339 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 10:06:12.740835 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":310,"threads_started":1}
I20260924 10:06:12.743633 31421 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37195
I20260924 10:06:12.743968 31421 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:12.744714 31421 heartbeater.cc:511] Master 127.13.162.62:37195 requested a full tablet report, sending...
I20260924 10:06:12.746116 31098 ts_manager.cc:194] Registered new tserver with Master: baeb1a7730ea4f04834517c49178708b (127.13.162.2:42991)
I20260924 10:06:12.746940 31098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:45147
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:06:12.775456 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37195
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:13.089707 31426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:13.090029 31426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:13.090168 31426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:13.097399 31426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:13.097522 31426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:13.097680 31426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:06:13.107252 31426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:13.108512 31426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:13.109977 31426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:13.122584 31431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:13.122522 31432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:13.122555 31434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:13.123593 31426 server_base.cc:1034] running on GCE node
I20260924 10:06:13.124145 31426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:13.125041 31426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:13.126260 31426 hybrid_clock.cc:648] HybridClock initialized: now 1790244373126203 us; error 46 us; skew 500 ppm
I20260924 10:06:13.126592 31426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:13 dist-test-slave-jvsj krb5kdc[31053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244373, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:13.129545 31426 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:06:13.130959 31426 webserver.cc:460] Webserver started at http://127.13.162.3:41317/ using document root <none> and password file <none>
I20260924 10:06:13.131637 31426 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:13.131742 31426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:13.132048 31426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:13.135007 31426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "17d5a9a16cd943fdbb64fb09ee274fd1"
format_stamp: "Formatted at 2026-09-24 10:06:13 on dist-test-slave-jvsj"
server_key: "f29868997d5be0f48a248399d224aa88"
server_key_iv: "8115925cfd85a2d251794bd72dc8b560"
server_key_version: "encryptionkey@0"
I20260924 10:06:13.135730 31426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "17d5a9a16cd943fdbb64fb09ee274fd1"
format_stamp: "Formatted at 2026-09-24 10:06:13 on dist-test-slave-jvsj"
server_key: "f29868997d5be0f48a248399d224aa88"
server_key_iv: "8115925cfd85a2d251794bd72dc8b560"
server_key_version: "encryptionkey@0"
I20260924 10:06:13.140698 31426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:13.144076 31441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:13.145465 31426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:06:13.145604 31426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "17d5a9a16cd943fdbb64fb09ee274fd1"
format_stamp: "Formatted at 2026-09-24 10:06:13 on dist-test-slave-jvsj"
server_key: "f29868997d5be0f48a248399d224aa88"
server_key_iv: "8115925cfd85a2d251794bd72dc8b560"
server_key_version: "encryptionkey@0"
I20260924 10:06:13.145776 31426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:13.175354 31426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:13.179468 31426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:13.179718 31426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:13.180644 31426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:13.181905 31426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:13.181973 31426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:13.182085 31426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:13.182142 31426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:13.211858 31426 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:45157
I20260924 10:06:13.211906 31553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:45157 every 8 connection(s)
I20260924 10:06:13.213136 31426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:06:13.216895 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31426
I20260924 10:06:13.217077 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:06:13.217638 13960 external_mini_cluster.cc:1428] Setting key d8b242b35771cadea00ea9b3f80e80a2
Sep 24 10:06:13 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244373, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:13.233564 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.216676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:38405 (local address 127.13.162.62:37195)
0924 10:06:13.216841 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.216846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.218372 (+ 1526us) server_negotiation.cc:407] Connection header received
0924 10:06:13.220321 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.220327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.220433 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.220516 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:13.221980 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.222837 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.224349 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.224759 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.224866 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.228134 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:13.228139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:13.228143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:13.228176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:13.230625 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.231320 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.231326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.231328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.231453 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.231851 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.231856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.231859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.232121 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:13.232252 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.232986 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.233215 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:06:13.234558 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.217389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37195 (local address 127.13.162.3:38405)
0924 10:06:13.218173 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.218224 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:13.220068 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:13.220753 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:13.220761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:13.221251 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:13.221810 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.221828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.222982 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.222986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.224205 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.224218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.224978 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.224985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.225213 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.225814 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:13.225843 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:13.227930 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:13.230757 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:13.230766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:13.230784 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:13.231206 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:13.231556 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:13.231561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:13.231565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:13.231747 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:13.232242 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:13.232250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:13.232635 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 10:06:13.232882 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 10:06:13.233438 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":318,"threads_started":1}
I20260924 10:06:13.235927 31554 heartbeater.cc:347] Connected to a master server at 127.13.162.62:37195
I20260924 10:06:13.236228 31554 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:13.236908 31554 heartbeater.cc:511] Master 127.13.162.62:37195 requested a full tablet report, sending...
I20260924 10:06:13.238345 31098 ts_manager.cc:194] Registered new tserver with Master: 17d5a9a16cd943fdbb64fb09ee274fd1 (127.13.162.3:45157)
I20260924 10:06:13.239439 31098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:38405
I20260924 10:06:13.244539 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:06:13.260892 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.249101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.13.162.62:37195)
0924 10:06:13.249296 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.249300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.249690 (+ 390us) server_negotiation.cc:407] Connection header received
0924 10:06:13.249835 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.249839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.249909 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.249986 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:13.251374 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.252088 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.253373 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.253766 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.253861 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.256221 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:13.256226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:13.256229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:13.256260 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:13.258648 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.259242 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.259246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.259248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.259392 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.259724 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.259730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.259733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.260040 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:13.260220 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.260521 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.260672 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 10:06:13.268503 31287 heartbeater.cc:503] Master 127.13.162.62:37195 was elected leader, sending a full tablet report...
I20260924 10:06:13.264942 31098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:06:13.272187 31098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:06:13.308068 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.299219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.13.162.3:45157)
0924 10:06:13.300819 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.300827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.301515 (+ 688us) server_negotiation.cc:407] Connection header received
0924 10:06:13.301738 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.301746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.301965 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.302149 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:13.302670 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.303809 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.305906 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.306960 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.307084 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.307335 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.307570 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.307728 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1435,"thread_start_us":1289,"threads_started":1}
I20260924 10:06:13.308121 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.298889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:45157 (local address 127.0.0.1:56590)
0924 10:06:13.301378 (+ 2489us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.301403 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:13.301549 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:13.302153 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:13.302157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:13.302182 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:13.302511 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.302522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.303953 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.303957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.305696 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.305717 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.307140 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.307146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.307329 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.307384 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 10:06:13.307512 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 10:06:13.307701 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2369,"thread_start_us":2206,"threads_started":1}
I20260924 10:06:13.308879 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.283819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39380)
0924 10:06:13.299515 (+ 15696us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.299539 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:13.299657 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:13.300803 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:13.300807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:13.300832 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:13.301598 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.301609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.303126 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.303132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.305869 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.305884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.308051 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.308057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.308231 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.308266 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:06:13.308346 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:06:13.308474 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15555,"mutex_wait_us":194,"thread_start_us":14078,"threads_started":1}
I20260924 10:06:13.309204 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.284548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.13.162.1:46259)
0924 10:06:13.300223 (+ 15675us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.300230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.300259 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:06:13.300366 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.300386 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.300624 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.300810 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:13.301765 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.303012 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.306050 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.307526 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.307656 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.307882 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.308652 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.308828 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15499,"thread_start_us":15349,"threads_started":1}
I20260924 10:06:13.309644 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.283880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.13.162.2:42991)
0924 10:06:13.297505 (+ 13625us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.297517 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.298829 (+ 1312us) server_negotiation.cc:407] Connection header received
0924 10:06:13.298956 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.298965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.300251 (+ 1286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.300469 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:13.301630 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.303068 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.306050 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.307526 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.307697 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.307855 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.309204 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.309358 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13416,"thread_start_us":13634,"threads_started":1}
I20260924 10:06:13.310135 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.283675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42991 (local address 127.0.0.1:52240)
0924 10:06:13.298347 (+ 14672us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.298399 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:13.298601 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:13.300745 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:13.300753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:13.300783 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:13.301461 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.301477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.303170 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.303179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.305880 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.305889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.308029 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.308041 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.308613 (+ 572us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.308645 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:06:13.308780 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:06:13.308925 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14351,"mutex_wait_us":219,"thread_start_us":16552,"threads_started":1}
I20260924 10:06:13.312824 31222 tablet_service.cc:1467] Processing CreateTablet for tablet 84bf371d583040f09c6e81cf217d9185 (DEFAULT_TABLE table=test-table [id=925a36f58f88429d993a4225af163d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:13.312587 31489 tablet_service.cc:1467] Processing CreateTablet for tablet 84bf371d583040f09c6e81cf217d9185 (DEFAULT_TABLE table=test-table [id=925a36f58f88429d993a4225af163d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:13.313007 31356 tablet_service.cc:1467] Processing CreateTablet for tablet 84bf371d583040f09c6e81cf217d9185 (DEFAULT_TABLE table=test-table [id=925a36f58f88429d993a4225af163d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:13.314126 31356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bf371d583040f09c6e81cf217d9185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:13.314162 31222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bf371d583040f09c6e81cf217d9185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:13.314157 31489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bf371d583040f09c6e81cf217d9185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:13.324996 31572 tablet_bootstrap.cc:492] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Bootstrap starting.
I20260924 10:06:13.328008 31573 tablet_bootstrap.cc:492] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Bootstrap starting.
I20260924 10:06:13.328306 31572 tablet_bootstrap.cc:654] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:13.329290 31574 tablet_bootstrap.cc:492] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Bootstrap starting.
I20260924 10:06:13.329689 31572 log.cc:824] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:13.331363 31573 tablet_bootstrap.cc:654] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:13.332363 31574 tablet_bootstrap.cc:654] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:13.332552 31573 log.cc:824] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:13.332885 31572 tablet_bootstrap.cc:492] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: No bootstrap required, opened a new log
I20260924 10:06:13.333164 31572 ts_tablet_manager.cc:1397] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260924 10:06:13.333437 31574 log.cc:824] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:13.336052 31573 tablet_bootstrap.cc:492] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: No bootstrap required, opened a new log
I20260924 10:06:13.336285 31574 tablet_bootstrap.cc:492] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: No bootstrap required, opened a new log
I20260924 10:06:13.336310 31573 ts_tablet_manager.cc:1397] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:06:13.336541 31574 ts_tablet_manager.cc:1397] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 10:06:13.342955 31573 raft_consensus.cc:348] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.343302 31573 raft_consensus.cc:374] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:13.343394 31573 raft_consensus.cc:729] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21abc50311284c039209ae7b3df50b73, State: Initialized, Role: FOLLOWER
I20260924 10:06:13.343314 31572 raft_consensus.cc:348] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.343762 31572 raft_consensus.cc:374] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:13.343865 31572 raft_consensus.cc:729] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baeb1a7730ea4f04834517c49178708b, State: Initialized, Role: FOLLOWER
I20260924 10:06:13.343963 31573 consensus_queue.cc:260] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.344406 31572 consensus_queue.cc:260] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.347044 31574 raft_consensus.cc:348] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.347419 31573 ts_tablet_manager.cc:1428] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 10:06:13.347464 31574 raft_consensus.cc:374] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:13.347549 31574 raft_consensus.cc:729] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d5a9a16cd943fdbb64fb09ee274fd1, State: Initialized, Role: FOLLOWER
I20260924 10:06:13.348127 31574 consensus_queue.cc:260] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.349365 31554 heartbeater.cc:503] Master 127.13.162.62:37195 was elected leader, sending a full tablet report...
I20260924 10:06:13.349884 31574 ts_tablet_manager.cc:1428] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 10:06:13.350777 31421 heartbeater.cc:503] Master 127.13.162.62:37195 was elected leader, sending a full tablet report...
I20260924 10:06:13.352524 31572 ts_tablet_manager.cc:1428] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
W20260924 10:06:13.465448 31555 tablet.cc:2369] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:13.472628 31422 tablet.cc:2369] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:13.489480 31288 tablet.cc:2369] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:13.555784 31578 raft_consensus.cc:482] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:06:13.556078 31578 raft_consensus.cc:504] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.557794 31578 leader_election.cc:288] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baeb1a7730ea4f04834517c49178708b (127.13.162.2:42991), 17d5a9a16cd943fdbb64fb09ee274fd1 (127.13.162.3:45157)
I20260924 10:06:13.564358 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.558421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:52221 (local address 127.13.162.3:45157)
0924 10:06:13.558620 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.558626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.559319 (+ 693us) server_negotiation.cc:407] Connection header received
0924 10:06:13.559469 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.559473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.559540 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.559628 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:13.560197 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.560941 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.562747 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.563496 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.563640 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.563717 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.563987 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.564113 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:06:13.564366 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.558298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:51931 (local address 127.13.162.2:42991)
0924 10:06:13.558547 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.558553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.559203 (+ 650us) server_negotiation.cc:407] Connection header received
0924 10:06:13.559365 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.559371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.559450 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.559519 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:13.560233 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.560916 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.562768 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.563522 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.563661 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.563729 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.564016 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.564136 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 10:06:13.564496 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.558078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:42991 (local address 127.13.162.1:51931)
0924 10:06:13.559082 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.559104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:13.559233 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:13.559549 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:13.559554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:13.559576 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:13.560101 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.560110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.561033 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.561038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.562632 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.562643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.563732 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.563738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.563873 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.563894 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:06:13.563977 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:06:13.564103 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":82,"spinlock_wait_cycles":36992,"thread_start_us":659,"threads_started":1}
I20260924 10:06:13.564798 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.558307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:45157 (local address 127.13.162.1:52221)
0924 10:06:13.559215 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.559232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:13.559353 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:13.559653 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:13.559658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:13.559690 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:13.560033 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.560045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.561062 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.561066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.562607 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:13.562621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.563696 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.563702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:13.563840 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.563863 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:06:13.563934 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:06:13.564049 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":42,"thread_start_us":484,"threads_started":1}
I20260924 10:06:13.565407 31376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bf371d583040f09c6e81cf217d9185" candidate_uuid: "21abc50311284c039209ae7b3df50b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baeb1a7730ea4f04834517c49178708b" is_pre_election: true
I20260924 10:06:13.565645 31509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bf371d583040f09c6e81cf217d9185" candidate_uuid: "21abc50311284c039209ae7b3df50b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" is_pre_election: true
I20260924 10:06:13.565869 31376 raft_consensus.cc:2454] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21abc50311284c039209ae7b3df50b73 in term 0.
I20260924 10:06:13.566082 31509 raft_consensus.cc:2454] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21abc50311284c039209ae7b3df50b73 in term 0.
I20260924 10:06:13.566542 31176 leader_election.cc:302] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21abc50311284c039209ae7b3df50b73, baeb1a7730ea4f04834517c49178708b; no voters:
I20260924 10:06:13.566958 31578 raft_consensus.cc:2781] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:06:13.567090 31578 raft_consensus.cc:482] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:06:13.567183 31578 raft_consensus.cc:3037] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:13.569360 31578 raft_consensus.cc:504] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.569919 31578 leader_election.cc:288] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [CANDIDATE]: Term 1 election: Requested vote from peers baeb1a7730ea4f04834517c49178708b (127.13.162.2:42991), 17d5a9a16cd943fdbb64fb09ee274fd1 (127.13.162.3:45157)
I20260924 10:06:13.570293 31376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bf371d583040f09c6e81cf217d9185" candidate_uuid: "21abc50311284c039209ae7b3df50b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baeb1a7730ea4f04834517c49178708b"
I20260924 10:06:13.570343 31509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bf371d583040f09c6e81cf217d9185" candidate_uuid: "21abc50311284c039209ae7b3df50b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1"
I20260924 10:06:13.570509 31376 raft_consensus.cc:3037] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:13.570519 31509 raft_consensus.cc:3037] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:13.572547 31376 raft_consensus.cc:2454] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21abc50311284c039209ae7b3df50b73 in term 1.
I20260924 10:06:13.572618 31509 raft_consensus.cc:2454] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21abc50311284c039209ae7b3df50b73 in term 1.
I20260924 10:06:13.572896 31176 leader_election.cc:302] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21abc50311284c039209ae7b3df50b73, baeb1a7730ea4f04834517c49178708b; no voters:
I20260924 10:06:13.573172 31578 raft_consensus.cc:2781] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:13.573572 31578 raft_consensus.cc:686] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 1 LEADER]: Becoming Leader. State: Replica: 21abc50311284c039209ae7b3df50b73, State: Running, Role: LEADER
I20260924 10:06:13.574003 31578 consensus_queue.cc:237] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } }
I20260924 10:06:13.578933 31098 catalog_manager.cc:5495] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21abc50311284c039209ae7b3df50b73 (127.13.162.1). New cstate: current_term: 1 leader_uuid: "21abc50311284c039209ae7b3df50b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21abc50311284c039209ae7b3df50b73" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 } health_report { overall_health: UNKNOWN } } }
I20260924 10:06:13.659968 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.654030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.13.162.1:46259)
0924 10:06:13.654222 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.654228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.654250 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:06:13.654461 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.654472 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.654581 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.654729 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:06:13.655037 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.656103 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.657490 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.658106 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.658219 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.658772 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:06:13.659457 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:06:13.659585 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.659630 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.659715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:06:13.668012 31376 raft_consensus.cc:1261] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 1 FOLLOWER]: Refusing update from remote peer 21abc50311284c039209ae7b3df50b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:13.668011 31509 raft_consensus.cc:1261] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 1 FOLLOWER]: Refusing update from remote peer 21abc50311284c039209ae7b3df50b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:13.668994 31583 consensus_queue.cc:1035] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baeb1a7730ea4f04834517c49178708b" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 42991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:13.669359 31578 consensus_queue.cc:1035] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d5a9a16cd943fdbb64fb09ee274fd1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:13.680135 31586 mvcc.cc:205] Tried to move back new op lower bound from 7332840954534383616 to 7332840954163027968. Current Snapshot: MvccSnapshot[applied={T|T < 7332840954534383616}]
I20260924 10:06:13.694500 31098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:13.694741 31098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:13.698243 31098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f2cbf6aea72415eb6aeafdb1d2e74ec: Sending DeleteTablet for 3 replicas of tablet 84bf371d583040f09c6e81cf217d9185
I20260924 10:06:13.699414 31356 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bf371d583040f09c6e81cf217d9185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:13 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:52240
I20260924 10:06:13.699710 31489 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bf371d583040f09c6e81cf217d9185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:13 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:56590
I20260924 10:06:13.699714 31222 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bf371d583040f09c6e81cf217d9185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:13 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:39380
I20260924 10:06:13.701699 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31159
I20260924 10:06:13.710580 31598 tablet_replica.cc:330] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: stopping tablet replica
I20260924 10:06:13.711076 31598 raft_consensus.cc:2229] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:06:13.711766 31598 raft_consensus.cc:2258] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:13.716246 31598 ts_tablet_manager.cc:1905] T 84bf371d583040f09c6e81cf217d9185 P 21abc50311284c039209ae7b3df50b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:13.718521 31596 tablet_replica.cc:330] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: stopping tablet replica
Sep 24 10:06:13 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:06:13.719224 31596 raft_consensus.cc:2229] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:13.720062 31596 raft_consensus.cc:2258] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:13.725011 31597 tablet_replica.cc:330] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: stopping tablet replica
I20260924 10:06:13.725533 31597 raft_consensus.cc:2229] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:13.726064 31597 raft_consensus.cc:2258] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:13.727361 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.703835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.13.162.1:46259)
0924 10:06:13.704049 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.704055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.707885 (+ 3830us) server_negotiation.cc:407] Connection header received
0924 10:06:13.708801 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.708808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.708905 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.708996 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:13.710923 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.712090 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.714180 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.714880 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.714976 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.720115 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:13.720128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:13.720145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:13.720201 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:13.724047 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.724930 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.724942 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.724946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.725108 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.725509 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.725514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.725518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.725938 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:13.726225 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.726787 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.727027 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 10:06:13.728667 31282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:13.730608 31597 ts_tablet_manager.cc:1905] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:13.731106 31596 ts_tablet_manager.cc:1905] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:13.737139 31597 ts_tablet_manager.cc:1918] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:13.737425 31597 log.cc:1196] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/84bf371d583040f09c6e81cf217d9185
I20260924 10:06:13.738081 31597 ts_tablet_manager.cc:1939] T 84bf371d583040f09c6e81cf217d9185 P 17d5a9a16cd943fdbb64fb09ee274fd1: Deleting consensus metadata
I20260924 10:06:13.738576 31596 ts_tablet_manager.cc:1918] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:13.738732 31596 log.cc:1196] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/84bf371d583040f09c6e81cf217d9185
I20260924 10:06:13.739241 31596 ts_tablet_manager.cc:1939] T 84bf371d583040f09c6e81cf217d9185 P baeb1a7730ea4f04834517c49178708b: Deleting consensus metadata
I20260924 10:06:13.740331 31084 catalog_manager.cc:4841] TS 17d5a9a16cd943fdbb64fb09ee274fd1 (127.13.162.3:45157): tablet 84bf371d583040f09c6e81cf217d9185 (table test-table [id=925a36f58f88429d993a4225af163d06]) successfully deleted
I20260924 10:06:13.740748 31083 catalog_manager.cc:4841] TS baeb1a7730ea4f04834517c49178708b (127.13.162.2:42991): tablet 84bf371d583040f09c6e81cf217d9185 (table test-table [id=925a36f58f88429d993a4225af163d06]) successfully deleted
I20260924 10:06:13.832796 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31159
W20260924 10:06:13.847589 31084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:46259 (error 108)
I20260924 10:06:13.847680 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31293
I20260924 10:06:13.849757 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.849036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39392)
0924 10:06:13.849434 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.849571 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":175,"threads_started":1}
W20260924 10:06:13.849977 31084 catalog_manager.cc:4584] TS 21abc50311284c039209ae7b3df50b73 (127.13.162.1:46259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84bf371d583040f09c6e81cf217d9185: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Sep 24 10:06:13 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:06:13.861574 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.848429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.13.162.2:42991)
0924 10:06:13.848628 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.848632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.848671 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:06:13.848836 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.848840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.848910 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.848989 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:13.850568 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.851283 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.852471 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.852859 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.852964 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.856937 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:13.856950 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:13.856965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:13.856998 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:13.859435 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.859993 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.859999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.860001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.860118 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.860415 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.860418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.860420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.860738 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:13.860951 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.861149 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.861299 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:06:13.862198 31416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:13.903569 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.903033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39402)
0924 10:06:13.903280 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.903396 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 10:06:13.904620 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.904217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39404)
0924 10:06:13.904384 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.904467 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:06:13.949182 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31293
I20260924 10:06:13.962679 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31426
I20260924 10:06:13.964933 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.964542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39412)
0924 10:06:13.964713 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.964786 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:06:13.965786 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.965462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39428)
0924 10:06:13.965596 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:13.965658 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
Sep 24 10:06:13 dist-test-slave-jvsj krb5kdc[31053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:06:13.976927 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:13.963496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.13.162.3:45157)
0924 10:06:13.963664 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:13.963669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:13.963690 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:06:13.963794 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:13.963799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:13.963862 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:13.963941 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:13.965221 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.965948 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.967376 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:13.967760 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:13.967855 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:13.972154 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:13.972170 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:13.972185 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:13.972215 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:13.974774 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.975355 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.975381 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.975383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.975519 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:13.975794 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:13.975797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:13.975799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:13.976111 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:13.976346 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:13.976542 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:06:13.976692 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:06:13.977520 31549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:14.046428 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:14.045899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39430)
0924 10:06:14.046115 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:14.046203 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 10:06:14.047418 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:14.047026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:46259 (local address 127.0.0.1:39438)
0924 10:06:14.047218 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:14.047284 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:46259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 10:06:14.064623 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31426
I20260924 10:06:14.077766 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31066
I20260924 10:06:14.078920 31128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:14.186110 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31066
2026-09-24T10:06:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4208 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T10:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:14Z Disabled control of system clock
I20260924 10:06:14.253567 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44759
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:06:14.573498 31609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:14.573781 31609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:14.573859 31609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:14.581032 31609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:14.581136 31609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:14.581173 31609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:14.581231 31609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:14.581274 31609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:14.590584 31609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:14.591886 31609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:14.593640 31609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:14.604549 31615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:14.604588 31617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:14.604622 31614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:14.605312 31609 server_base.cc:1034] running on GCE node
I20260924 10:06:14.606005 31609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:14.607388 31609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:14.608597 31609 hybrid_clock.cc:648] HybridClock initialized: now 1790244374608542 us; error 48 us; skew 500 ppm
I20260924 10:06:14.608922 31609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:14.610994 31609 webserver.cc:460] Webserver started at http://127.13.162.62:44301/ using document root <none> and password file <none>
I20260924 10:06:14.611660 31609 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:14.611760 31609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:14.612053 31609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:14.614840 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "d67b14ed73c04faaad909efe4df2a5a6"
format_stamp: "Formatted at 2026-09-24 10:06:14 on dist-test-slave-jvsj"
server_key: "950f8bcf878549e755e8d24363297bce"
server_key_iv: "a850e8a0246669db9b7a228ffb5c8a10"
server_key_version: "encryptionkey@0"
I20260924 10:06:14.615600 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "d67b14ed73c04faaad909efe4df2a5a6"
format_stamp: "Formatted at 2026-09-24 10:06:14 on dist-test-slave-jvsj"
server_key: "950f8bcf878549e755e8d24363297bce"
server_key_iv: "a850e8a0246669db9b7a228ffb5c8a10"
server_key_version: "encryptionkey@0"
I20260924 10:06:14.620447 31609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:06:14.623728 31623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:14.625030 31609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:06:14.625193 31609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "d67b14ed73c04faaad909efe4df2a5a6"
format_stamp: "Formatted at 2026-09-24 10:06:14 on dist-test-slave-jvsj"
server_key: "950f8bcf878549e755e8d24363297bce"
server_key_iv: "a850e8a0246669db9b7a228ffb5c8a10"
server_key_version: "encryptionkey@0"
I20260924 10:06:14.625393 31609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:14.649305 31609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:14.650158 31609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:14.650466 31609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:14.667580 31609 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:44759
I20260924 10:06:14.667606 31674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:44759 every 8 connection(s)
I20260924 10:06:14.668937 31609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:14.672802 31675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:14.673313 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31609
I20260924 10:06:14.673478 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:14.673918 13960 external_mini_cluster.cc:1428] Setting key bf25a1e5adaf63cd7fc2f869490351e4
I20260924 10:06:14.685719 31675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6: Bootstrap starting.
I20260924 10:06:14.687348 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:14.675819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.13.162.62:44759)
0924 10:06:14.676815 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:14.676826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:14.676869 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:06:14.678639 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:14.678661 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:14.679131 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:14.679551 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:14.680049 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:14.681306 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:14.683202 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:14.683900 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:14.684021 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:14.685205 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:14.685212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:14.685224 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:14.685724 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:14.685796 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:14.685821 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:14.685926 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:14.686102 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 10:06:14.686423 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":380,"threads_started":1}
I20260924 10:06:14.689123 31675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:14.690199 31675 log.cc:824] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:14.692796 31675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6: No bootstrap required, opened a new log
I20260924 10:06:14.700503 31675 raft_consensus.cc:348] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67b14ed73c04faaad909efe4df2a5a6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44759 } }
I20260924 10:06:14.700865 31675 raft_consensus.cc:374] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:14.700974 31675 raft_consensus.cc:729] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b14ed73c04faaad909efe4df2a5a6, State: Initialized, Role: FOLLOWER
I20260924 10:06:14.701485 31675 consensus_queue.cc:260] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67b14ed73c04faaad909efe4df2a5a6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44759 } }
I20260924 10:06:14.701695 31675 raft_consensus.cc:388] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:14.701815 31675 raft_consensus.cc:482] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:14.701958 31675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:14.704048 31675 raft_consensus.cc:504] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67b14ed73c04faaad909efe4df2a5a6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44759 } }
I20260924 10:06:14.704504 31675 leader_election.cc:302] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d67b14ed73c04faaad909efe4df2a5a6; no voters:
I20260924 10:06:14.705037 31675 leader_election.cc:288] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:14.705183 31680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:14.705825 31680 raft_consensus.cc:686] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [term 1 LEADER]: Becoming Leader. State: Replica: d67b14ed73c04faaad909efe4df2a5a6, State: Running, Role: LEADER
I20260924 10:06:14.706480 31680 consensus_queue.cc:237] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67b14ed73c04faaad909efe4df2a5a6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44759 } }
I20260924 10:06:14.707794 31675 sys_catalog.cc:564] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:14.709132 31682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d67b14ed73c04faaad909efe4df2a5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67b14ed73c04faaad909efe4df2a5a6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44759 } } }
I20260924 10:06:14.709110 31681 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d67b14ed73c04faaad909efe4df2a5a6. Latest consensus state: current_term: 1 leader_uuid: "d67b14ed73c04faaad909efe4df2a5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d67b14ed73c04faaad909efe4df2a5a6" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44759 } } }
I20260924 10:06:14.709566 31681 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:14.709570 31682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:14.711494 31685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:14.717306 31685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:14.726125 31685 catalog_manager.cc:1347] Generated new cluster ID: df0ed98548af4a9ca35bfe93476f477f
I20260924 10:06:14.726245 31685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:14.752516 31685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:14.753799 31685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:14.767309 31685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6: Generated new TSK 0
I20260924 10:06:14.768126 31685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:14.778026 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44759
--builtin_ntp_servers=127.13.162.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:06:15.109434 31699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:15.109704 31699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:15.109826 31699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:15.117241 31699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:15.117344 31699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:15.117486 31699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:06:15.126637 31699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:15.127902 31699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:15.129706 31699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:15.142010 31704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:15.142483 31707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:15.142540 31705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:15.143013 31699 server_base.cc:1034] running on GCE node
I20260924 10:06:15.143540 31699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:15.144397 31699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:15.145594 31699 hybrid_clock.cc:648] HybridClock initialized: now 1790244375145555 us; error 32 us; skew 500 ppm
I20260924 10:06:15.145887 31699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:15.151573 31699 webserver.cc:460] Webserver started at http://127.13.162.1:41671/ using document root <none> and password file <none>
I20260924 10:06:15.152308 31699 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:15.152407 31699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:15.152714 31699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:15.155642 31699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "ffff7ebd350841efaeb03d588edfcd6e"
format_stamp: "Formatted at 2026-09-24 10:06:15 on dist-test-slave-jvsj"
server_key: "d0b3daa446a70787d3d62b462da7d73b"
server_key_iv: "2d93a0b03782d8594cad9b79924bac24"
server_key_version: "encryptionkey@0"
I20260924 10:06:15.156410 31699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "ffff7ebd350841efaeb03d588edfcd6e"
format_stamp: "Formatted at 2026-09-24 10:06:15 on dist-test-slave-jvsj"
server_key: "d0b3daa446a70787d3d62b462da7d73b"
server_key_iv: "2d93a0b03782d8594cad9b79924bac24"
server_key_version: "encryptionkey@0"
I20260924 10:06:15.161290 31699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:06:15.164582 31713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:15.165866 31699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:06:15.166015 31699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "ffff7ebd350841efaeb03d588edfcd6e"
format_stamp: "Formatted at 2026-09-24 10:06:15 on dist-test-slave-jvsj"
server_key: "d0b3daa446a70787d3d62b462da7d73b"
server_key_iv: "2d93a0b03782d8594cad9b79924bac24"
server_key_version: "encryptionkey@0"
I20260924 10:06:15.166201 31699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:15.185946 31699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:15.186813 31699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:15.187093 31699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:15.188078 31699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:15.189522 31699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:15.189617 31699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:15.189718 31699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:15.189785 31699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:15.218823 31699 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:38183
I20260924 10:06:15.218901 31825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:38183 every 8 connection(s)
I20260924 10:06:15.220352 31699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:06:15.222880 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31699
I20260924 10:06:15.223043 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:06:15.223485 13960 external_mini_cluster.cc:1428] Setting key fa99f08e6c8d2dadf9fc016c078dfd11
I20260924 10:06:15.226414 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44759
--builtin_ntp_servers=127.13.162.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:06:15.233579 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:15.224080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:44383 (local address 127.13.162.62:44759)
0924 10:06:15.225438 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:15.225442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:15.225458 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:06:15.226845 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:15.226850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:15.226927 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:15.227040 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:15.228074 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.229066 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.230755 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.231324 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.231456 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:15.232780 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:15.232787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:15.232792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:15.232816 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:15.232860 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:15.232876 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:15.232961 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:15.233173 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 10:06:15.233273 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":1162,"threads_started":1}
I20260924 10:06:15.234788 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:15.223724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44759 (local address 127.13.162.1:44383)
0924 10:06:15.224758 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:15.224810 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:15.226573 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:15.227229 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:15.227237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:15.227330 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:15.227914 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:15.227939 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.229191 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.229195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:15.230631 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:15.230643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.231559 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.231565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:15.231775 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:15.232469 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:15.232472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:15.232513 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:15.232596 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:15.232986 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:15.232996 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:15.233002 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:06:15.233147 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:06:15.233514 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260924 10:06:15.236349 31826 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44759
I20260924 10:06:15.236660 31826 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:15.237514 31826 heartbeater.cc:511] Master 127.13.162.62:44759 requested a full tablet report, sending...
I20260924 10:06:15.239563 31640 ts_manager.cc:194] Registered new tserver with Master: ffff7ebd350841efaeb03d588edfcd6e (127.13.162.1:38183)
I20260924 10:06:15.240995 31640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.1:44383
W20260924 10:06:15.543112 31830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:15.543407 31830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:15.543537 31830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:15.551294 31830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:15.551416 31830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:15.551580 31830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:06:15.560760 31830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:15.562019 31830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:15.563831 31830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:15.575934 31835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:15.575963 31836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:15.576076 31838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:15.576625 31830 server_base.cc:1034] running on GCE node
I20260924 10:06:15.577195 31830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:15.578069 31830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:15.579267 31830 hybrid_clock.cc:648] HybridClock initialized: now 1790244375579214 us; error 40 us; skew 500 ppm
I20260924 10:06:15.579597 31830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:15.581707 31830 webserver.cc:460] Webserver started at http://127.13.162.2:40907/ using document root <none> and password file <none>
I20260924 10:06:15.582398 31830 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:15.582500 31830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:15.582788 31830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:15.585650 31830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "c1db29ef578a4f70b2d8c145d5d27f1a"
format_stamp: "Formatted at 2026-09-24 10:06:15 on dist-test-slave-jvsj"
server_key: "567871f9d16534c202e24ebf876d9136"
server_key_iv: "ac2a83fecbbf3fbbe84a03557877459a"
server_key_version: "encryptionkey@0"
I20260924 10:06:15.586491 31830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "c1db29ef578a4f70b2d8c145d5d27f1a"
format_stamp: "Formatted at 2026-09-24 10:06:15 on dist-test-slave-jvsj"
server_key: "567871f9d16534c202e24ebf876d9136"
server_key_iv: "ac2a83fecbbf3fbbe84a03557877459a"
server_key_version: "encryptionkey@0"
I20260924 10:06:15.591667 31830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:06:15.595211 31844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:15.596678 31830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:15.596843 31830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "c1db29ef578a4f70b2d8c145d5d27f1a"
format_stamp: "Formatted at 2026-09-24 10:06:15 on dist-test-slave-jvsj"
server_key: "567871f9d16534c202e24ebf876d9136"
server_key_iv: "ac2a83fecbbf3fbbe84a03557877459a"
server_key_version: "encryptionkey@0"
I20260924 10:06:15.597039 31830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:15.611987 31830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:15.612748 31830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:15.613008 31830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:15.613891 31830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:15.615258 31830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:15.615365 31830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:15.615463 31830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:15.615535 31830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:15.644865 31830 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:32951
I20260924 10:06:15.644909 31956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:32951 every 8 connection(s)
I20260924 10:06:15.646122 31830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:06:15.650264 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31830
I20260924 10:06:15.650447 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:06:15.650866 13960 external_mini_cluster.cc:1428] Setting key 7c525bd3fb4f1ee828c86495ad47bb1c
I20260924 10:06:15.653687 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44759
--builtin_ntp_servers=127.13.162.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:06:15.658741 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:15.648541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:43413 (local address 127.13.162.62:44759)
0924 10:06:15.648714 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:15.648718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:15.650188 (+ 1470us) server_negotiation.cc:407] Connection header received
0924 10:06:15.652177 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:15.652182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:15.652245 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:15.652382 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:15.653519 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.654519 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.656047 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.656613 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.656726 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:15.657954 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:15.657960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:15.657965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:15.657986 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:15.658022 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:15.658036 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:15.658112 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:15.658303 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:06:15.658453 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:06:15.659749 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:15.649135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44759 (local address 127.13.162.2:43413)
0924 10:06:15.650033 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:15.650067 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:15.651877 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:15.652623 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:15.652634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:15.652752 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:15.653354 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:15.653371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.654659 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.654662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:15.655898 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:15.655910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:15.656806 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:15.656811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:15.657022 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:15.657660 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:15.657663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:15.657702 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:15.657779 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:15.658122 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:15.658128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:15.658133 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:06:15.658269 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 10:06:15.658706 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":275,"threads_started":1}
I20260924 10:06:15.661093 31957 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44759
I20260924 10:06:15.661374 31957 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:15.662177 31957 heartbeater.cc:511] Master 127.13.162.62:44759 requested a full tablet report, sending...
I20260924 10:06:15.663551 31640 ts_manager.cc:194] Registered new tserver with Master: c1db29ef578a4f70b2d8c145d5d27f1a (127.13.162.2:32951)
I20260924 10:06:15.664325 31640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.2:43413
W20260924 10:06:15.969344 31960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:15.969627 31960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:15.969748 31960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:15.977144 31960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:15.977241 31960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:15.977375 31960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:06:15.986634 31960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:15.987959 31960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:15.989801 31960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:16.002377 31966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:16.002408 31965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:16.002661 31968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:16.003479 31960 server_base.cc:1034] running on GCE node
I20260924 10:06:16.004009 31960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:16.004874 31960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:16.006065 31960 hybrid_clock.cc:648] HybridClock initialized: now 1790244376006005 us; error 46 us; skew 500 ppm
I20260924 10:06:16.006443 31960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:16.008561 31960 webserver.cc:460] Webserver started at http://127.13.162.3:37921/ using document root <none> and password file <none>
I20260924 10:06:16.009223 31960 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:16.009311 31960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:16.009626 31960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:16.012449 31960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "de595dd5ee7344a8aa517df816c9ad7e"
format_stamp: "Formatted at 2026-09-24 10:06:16 on dist-test-slave-jvsj"
server_key: "6f173e1c70d9266965eb84ea0e93a6d5"
server_key_iv: "b846c3dfcd60e0a41cce42158126f4f1"
server_key_version: "encryptionkey@0"
I20260924 10:06:16.013165 31960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "de595dd5ee7344a8aa517df816c9ad7e"
format_stamp: "Formatted at 2026-09-24 10:06:16 on dist-test-slave-jvsj"
server_key: "6f173e1c70d9266965eb84ea0e93a6d5"
server_key_iv: "b846c3dfcd60e0a41cce42158126f4f1"
server_key_version: "encryptionkey@0"
I20260924 10:06:16.018272 31960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:06:16.021819 31974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:16.023226 31960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:06:16.023360 31960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "de595dd5ee7344a8aa517df816c9ad7e"
format_stamp: "Formatted at 2026-09-24 10:06:16 on dist-test-slave-jvsj"
server_key: "6f173e1c70d9266965eb84ea0e93a6d5"
server_key_iv: "b846c3dfcd60e0a41cce42158126f4f1"
server_key_version: "encryptionkey@0"
I20260924 10:06:16.023573 31960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:16.077466 31960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:16.078310 31960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:16.078595 31960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:16.079557 31960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:16.080906 31960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:16.080987 31960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:16.081064 31960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:16.081113 31960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:16.112577 31960 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:38467
I20260924 10:06:16.112628 32086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:38467 every 8 connection(s)
I20260924 10:06:16.113847 31960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:06:16.117218 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 31960
I20260924 10:06:16.117375 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:06:16.117813 13960 external_mini_cluster.cc:1428] Setting key 453d14365af30c434fc1aec024b98cff
I20260924 10:06:16.125869 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.116340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:45923 (local address 127.13.162.62:44759)
0924 10:06:16.116551 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.116557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.118109 (+ 1552us) server_negotiation.cc:407] Connection header received
0924 10:06:16.119878 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.119884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.119973 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.120056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:16.121104 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.121828 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.123394 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.123800 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.123904 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.125103 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:16.125108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:16.125112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:16.125128 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:16.125157 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:16.125169 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:16.125239 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.125437 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.125534 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:06:16.126894 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.116952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44759 (local address 127.13.162.3:45923)
0924 10:06:16.117948 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.117992 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:16.119616 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:16.120294 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:16.120303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:16.120398 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:16.120952 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.120968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.121978 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.121982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.123259 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.123271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.124000 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.124007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.124226 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.124805 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:16.124808 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:16.124846 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:16.124928 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:16.125251 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:16.125262 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:16.125270 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:06:16.125415 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:06:16.125852 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":292,"threads_started":1}
I20260924 10:06:16.128276 32087 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44759
I20260924 10:06:16.128554 32087 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:16.129274 32087 heartbeater.cc:511] Master 127.13.162.62:44759 requested a full tablet report, sending...
I20260924 10:06:16.130723 31640 ts_manager.cc:194] Registered new tserver with Master: de595dd5ee7344a8aa517df816c9ad7e (127.13.162.3:38467)
I20260924 10:06:16.131610 31640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.3:45923
I20260924 10:06:16.132496 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:06:16.142433 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.136853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.13.162.62:44759)
0924 10:06:16.137027 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.137033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.137419 (+ 386us) server_negotiation.cc:407] Connection header received
0924 10:06:16.137615 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.137621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.137699 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.137777 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:16.138248 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.138996 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.140286 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.140751 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.140853 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.141752 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:16.141759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:16.141762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:16.141777 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:16.141806 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:16.141817 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:16.141886 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.142011 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.142111 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:06:16.145859 31640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:06:16.152467 31640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:06:16.196573 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.163985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.13.162.2:32951)
0924 10:06:16.176588 (+ 12603us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.176600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.186224 (+ 9624us) server_negotiation.cc:407] Connection header received
0924 10:06:16.187916 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.187936 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.188271 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.188502 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:16.189556 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.191117 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.193756 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.195059 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.195184 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.195435 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.196049 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.196231 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12432,"thread_start_us":12318,"threads_started":1}
I20260924 10:06:16.196604 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.164380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58266)
0924 10:06:16.188362 (+ 23982us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.188378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:16.188503 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:16.189383 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:16.189387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:16.189412 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:16.189780 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.189789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.191078 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.191082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.193955 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.193974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.195725 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.195732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.195882 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.195912 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:06:16.195996 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:06:16.196208 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23873,"mutex_wait_us":342,"thread_start_us":22761,"threads_started":1}
I20260924 10:06:16.197202 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.164162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:38467 (local address 127.0.0.1:39088)
0924 10:06:16.187403 (+ 23241us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.187424 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:16.187538 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:16.188994 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:16.188998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:16.189037 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:16.189508 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.189524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.191426 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.191431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.194022 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.194033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.196027 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.196034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.196214 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.196242 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:06:16.196345 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:06:16.196483 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23104,"mutex_wait_us":145,"thread_start_us":23961,"threads_started":1}
I20260924 10:06:16.197252 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.163846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:32951 (local address 127.0.0.1:49452)
0924 10:06:16.186029 (+ 22183us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.186081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:16.187207 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:16.188849 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:16.188855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:16.188892 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:16.189381 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.189399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.191248 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.191253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.193595 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.193607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.195569 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.195576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.195778 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.195827 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 10:06:16.196779 (+ 952us) client_negotiation.cc:240] Negotiation successful
0924 10:06:16.196893 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21914,"mutex_wait_us":283,"spinlock_wait_cycles":12544,"thread_start_us":21742,"threads_started":1}
I20260924 10:06:16.197717 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.164811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.13.162.3:38467)
0924 10:06:16.188114 (+ 23303us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.188123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.188164 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:06:16.188264 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.188271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.188536 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.188717 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:16.189720 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.191261 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.194196 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.195548 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.195668 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.195907 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.196406 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.196578 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23136,"thread_start_us":22968,"threads_started":1}
I20260924 10:06:16.198096 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.164851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.13.162.1:38183)
0924 10:06:16.186978 (+ 22127us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.186987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.188629 (+ 1642us) server_negotiation.cc:407] Connection header received
0924 10:06:16.188728 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.188735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.188973 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.189152 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:16.189925 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.190938 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.194195 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.195231 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.195350 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.195600 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.197628 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.197807 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21933,"thread_start_us":21804,"threads_started":1}
I20260924 10:06:16.201165 31892 tablet_service.cc:1467] Processing CreateTablet for tablet a9f50c0a300347afaa4b0c3324017cfc (DEFAULT_TABLE table=test-table [id=7ed085a63362424c80f76a5630a6b9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:16.201501 31761 tablet_service.cc:1467] Processing CreateTablet for tablet a9f50c0a300347afaa4b0c3324017cfc (DEFAULT_TABLE table=test-table [id=7ed085a63362424c80f76a5630a6b9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:16.201491 32022 tablet_service.cc:1467] Processing CreateTablet for tablet a9f50c0a300347afaa4b0c3324017cfc (DEFAULT_TABLE table=test-table [id=7ed085a63362424c80f76a5630a6b9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:16.202471 31892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f50c0a300347afaa4b0c3324017cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:16.202785 32022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f50c0a300347afaa4b0c3324017cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:16.202785 31761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f50c0a300347afaa4b0c3324017cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:16.213640 32105 tablet_bootstrap.cc:492] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Bootstrap starting.
I20260924 10:06:16.217139 32107 tablet_bootstrap.cc:492] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Bootstrap starting.
I20260924 10:06:16.217551 32106 tablet_bootstrap.cc:492] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Bootstrap starting.
I20260924 10:06:16.217258 32105 tablet_bootstrap.cc:654] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:16.219139 32105 log.cc:824] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:16.221128 32107 tablet_bootstrap.cc:654] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:16.222371 32107 log.cc:824] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:16.226076 32105 tablet_bootstrap.cc:492] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: No bootstrap required, opened a new log
I20260924 10:06:16.226447 32105 ts_tablet_manager.cc:1397] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260924 10:06:16.227742 32107 tablet_bootstrap.cc:492] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: No bootstrap required, opened a new log
I20260924 10:06:16.228093 32107 ts_tablet_manager.cc:1397] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 10:06:16.234980 32106 tablet_bootstrap.cc:654] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:16.236260 32106 log.cc:824] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:16.236835 32105 raft_consensus.cc:348] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.237310 32105 raft_consensus.cc:374] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:16.237396 32105 raft_consensus.cc:729] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1db29ef578a4f70b2d8c145d5d27f1a, State: Initialized, Role: FOLLOWER
I20260924 10:06:16.238137 32105 consensus_queue.cc:260] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.238257 32107 raft_consensus.cc:348] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.238719 32107 raft_consensus.cc:374] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:16.238821 32107 raft_consensus.cc:729] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de595dd5ee7344a8aa517df816c9ad7e, State: Initialized, Role: FOLLOWER
I20260924 10:06:16.239549 32107 consensus_queue.cc:260] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.239826 31957 heartbeater.cc:503] Master 127.13.162.62:44759 was elected leader, sending a full tablet report...
I20260924 10:06:16.240571 32087 heartbeater.cc:503] Master 127.13.162.62:44759 was elected leader, sending a full tablet report...
I20260924 10:06:16.240594 32105 ts_tablet_manager.cc:1428] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
I20260924 10:06:16.241176 32107 ts_tablet_manager.cc:1428] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 10:06:16.242775 32106 tablet_bootstrap.cc:492] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: No bootstrap required, opened a new log
I20260924 10:06:16.243104 32106 ts_tablet_manager.cc:1397] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20260924 10:06:16.244383 31826 heartbeater.cc:503] Master 127.13.162.62:44759 was elected leader, sending a full tablet report...
I20260924 10:06:16.251299 32106 raft_consensus.cc:348] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.251796 32106 raft_consensus.cc:374] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:16.251899 32106 raft_consensus.cc:729] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffff7ebd350841efaeb03d588edfcd6e, State: Initialized, Role: FOLLOWER
I20260924 10:06:16.252470 32106 consensus_queue.cc:260] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.253927 32106 ts_tablet_manager.cc:1428] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 10:06:16.267870 32113 raft_consensus.cc:482] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:06:16.268172 32113 raft_consensus.cc:504] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.270047 32113 leader_election.cc:288] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1db29ef578a4f70b2d8c145d5d27f1a (127.13.162.2:32951), de595dd5ee7344a8aa517df816c9ad7e (127.13.162.3:38467)
I20260924 10:06:16.276798 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.270566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:48543 (local address 127.13.162.2:32951)
0924 10:06:16.270784 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.270791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.271508 (+ 717us) server_negotiation.cc:407] Connection header received
0924 10:06:16.271688 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.271692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.271771 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.271844 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:16.272420 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.273109 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.275202 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.275918 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.276073 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.276145 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.276407 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.276549 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:06:16.276844 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.270641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:35477 (local address 127.13.162.3:38467)
0924 10:06:16.270829 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.270836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.271581 (+ 745us) server_negotiation.cc:407] Connection header received
0924 10:06:16.271738 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.271742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.271799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.271872 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:16.272454 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.273147 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.275208 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.275994 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.276145 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.276211 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.276472 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.276603 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:06:16.276932 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.270416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:32951 (local address 127.13.162.1:48543)
0924 10:06:16.271400 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.271416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:16.271560 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:16.271872 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:16.271877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:16.271899 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:16.272283 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.272296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.273228 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.273234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.274991 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.275009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.276089 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.276095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.276247 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.276271 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:06:16.276362 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:06:16.276504 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":92,"thread_start_us":579,"threads_started":1}
I20260924 10:06:16.277144 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.270733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:38467 (local address 127.13.162.1:35477)
0924 10:06:16.271476 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.271490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:16.271614 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:16.271902 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:16.271906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:16.271927 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:16.272313 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.272321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.273281 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.273284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.275002 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:16.275014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.276180 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.276187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:16.276333 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.276356 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:06:16.276429 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:06:16.276575 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":116,"thread_start_us":405,"threads_started":1}
I20260924 10:06:16.278023 31912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f50c0a300347afaa4b0c3324017cfc" candidate_uuid: "ffff7ebd350841efaeb03d588edfcd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" is_pre_election: true
I20260924 10:06:16.278054 32042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f50c0a300347afaa4b0c3324017cfc" candidate_uuid: "ffff7ebd350841efaeb03d588edfcd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de595dd5ee7344a8aa517df816c9ad7e" is_pre_election: true
I20260924 10:06:16.278640 31912 raft_consensus.cc:2454] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffff7ebd350841efaeb03d588edfcd6e in term 0.
I20260924 10:06:16.278640 32042 raft_consensus.cc:2454] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffff7ebd350841efaeb03d588edfcd6e in term 0.
I20260924 10:06:16.279326 31715 leader_election.cc:302] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1db29ef578a4f70b2d8c145d5d27f1a, ffff7ebd350841efaeb03d588edfcd6e; no voters:
I20260924 10:06:16.279750 32113 raft_consensus.cc:2781] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:06:16.279894 32113 raft_consensus.cc:482] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:06:16.279981 32113 raft_consensus.cc:3037] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:16.282087 32113 raft_consensus.cc:504] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.282727 32113 leader_election.cc:288] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [CANDIDATE]: Term 1 election: Requested vote from peers c1db29ef578a4f70b2d8c145d5d27f1a (127.13.162.2:32951), de595dd5ee7344a8aa517df816c9ad7e (127.13.162.3:38467)
I20260924 10:06:16.283246 32042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f50c0a300347afaa4b0c3324017cfc" candidate_uuid: "ffff7ebd350841efaeb03d588edfcd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de595dd5ee7344a8aa517df816c9ad7e"
I20260924 10:06:16.283246 31912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f50c0a300347afaa4b0c3324017cfc" candidate_uuid: "ffff7ebd350841efaeb03d588edfcd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a"
I20260924 10:06:16.283458 32042 raft_consensus.cc:3037] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:16.283468 31912 raft_consensus.cc:3037] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:16.286610 32042 raft_consensus.cc:2454] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffff7ebd350841efaeb03d588edfcd6e in term 1.
I20260924 10:06:16.286610 31912 raft_consensus.cc:2454] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffff7ebd350841efaeb03d588edfcd6e in term 1.
I20260924 10:06:16.287055 31717 leader_election.cc:302] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de595dd5ee7344a8aa517df816c9ad7e, ffff7ebd350841efaeb03d588edfcd6e; no voters:
I20260924 10:06:16.287458 32113 raft_consensus.cc:2781] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:16.287912 32113 raft_consensus.cc:686] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 1 LEADER]: Becoming Leader. State: Replica: ffff7ebd350841efaeb03d588edfcd6e, State: Running, Role: LEADER
I20260924 10:06:16.288408 32113 consensus_queue.cc:237] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } }
I20260924 10:06:16.292590 31639 catalog_manager.cc:5495] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to ffff7ebd350841efaeb03d588edfcd6e (127.13.162.1). New cstate: current_term: 1 leader_uuid: "ffff7ebd350841efaeb03d588edfcd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffff7ebd350841efaeb03d588edfcd6e" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 38183 } health_report { overall_health: HEALTHY } } }
I20260924 10:06:16.326069 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.321149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.13.162.1:38183)
0924 10:06:16.321331 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.321335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.321353 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:06:16.321480 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.321484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.321578 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.321698 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:06:16.322000 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.322709 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.324196 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.324599 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.324693 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.325106 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:06:16.325592 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:06:16.325702 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.325741 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.325803 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:06:16.334151 32042 raft_consensus.cc:1261] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 1 FOLLOWER]: Refusing update from remote peer ffff7ebd350841efaeb03d588edfcd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:16.334157 31912 raft_consensus.cc:1261] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 1 FOLLOWER]: Refusing update from remote peer ffff7ebd350841efaeb03d588edfcd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:16.335070 32113 consensus_queue.cc:1035] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1db29ef578a4f70b2d8c145d5d27f1a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:16.335361 32116 consensus_queue.cc:1035] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de595dd5ee7344a8aa517df816c9ad7e" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:16.346010 32119 mvcc.cc:205] Tried to move back new op lower bound from 7332840965455147008 to 7332840965280727040. Current Snapshot: MvccSnapshot[applied={T|T < 7332840965455147008}]
I20260924 10:06:16.359004 31639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:16.359261 31639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:16.362748 31639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d67b14ed73c04faaad909efe4df2a5a6: Sending DeleteTablet for 3 replicas of tablet a9f50c0a300347afaa4b0c3324017cfc
I20260924 10:06:16.363905 31892 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f50c0a300347afaa4b0c3324017cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:16 UTC) from {username='slave'} at 127.0.0.1:49452
I20260924 10:06:16.364267 31761 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f50c0a300347afaa4b0c3324017cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:16 UTC) from {username='slave'} at 127.0.0.1:58266
I20260924 10:06:16.364279 32022 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f50c0a300347afaa4b0c3324017cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:16 UTC) from {username='slave'} at 127.0.0.1:39088
I20260924 10:06:16.365918 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31699
I20260924 10:06:16.376417 32130 tablet_replica.cc:330] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: stopping tablet replica
I20260924 10:06:16.376943 32130 raft_consensus.cc:2229] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:06:16.377766 32130 raft_consensus.cc:2258] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:16.378545 32131 tablet_replica.cc:330] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: stopping tablet replica
I20260924 10:06:16.378861 32131 raft_consensus.cc:2229] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:16.379267 32131 raft_consensus.cc:2258] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:16.381642 32129 tablet_replica.cc:330] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: stopping tablet replica
I20260924 10:06:16.382031 32129 raft_consensus.cc:2229] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:16.382532 32129 raft_consensus.cc:2258] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:16.383468 32130 ts_tablet_manager.cc:1905] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:16.384608 32129 ts_tablet_manager.cc:1905] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:16.389168 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.367320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.13.162.1:38183)
0924 10:06:16.367500 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.367506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.380807 (+ 13301us) server_negotiation.cc:407] Connection header received
0924 10:06:16.380998 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.381003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.381109 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.381234 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:16.381978 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.383072 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.385122 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.385765 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.385866 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.387316 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:16.387326 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:16.387342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:16.388178 (+ 836us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:16.388241 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:16.388269 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:16.388362 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.388626 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.388740 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:06:16.390611 31821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:16.391377 32129 ts_tablet_manager.cc:1918] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:16.391592 32129 log.cc:1196] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/a9f50c0a300347afaa4b0c3324017cfc
I20260924 10:06:16.391952 32131 ts_tablet_manager.cc:1905] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:16.392107 32129 ts_tablet_manager.cc:1939] T a9f50c0a300347afaa4b0c3324017cfc P c1db29ef578a4f70b2d8c145d5d27f1a: Deleting consensus metadata
I20260924 10:06:16.394721 31625 catalog_manager.cc:4841] TS c1db29ef578a4f70b2d8c145d5d27f1a (127.13.162.2:32951): tablet a9f50c0a300347afaa4b0c3324017cfc (table test-table [id=7ed085a63362424c80f76a5630a6b9ba]) successfully deleted
I20260924 10:06:16.399789 32131 ts_tablet_manager.cc:1918] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:16.399967 32131 log.cc:1196] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/a9f50c0a300347afaa4b0c3324017cfc
I20260924 10:06:16.400463 32131 ts_tablet_manager.cc:1939] T a9f50c0a300347afaa4b0c3324017cfc P de595dd5ee7344a8aa517df816c9ad7e: Deleting consensus metadata
I20260924 10:06:16.402298 31627 catalog_manager.cc:4841] TS de595dd5ee7344a8aa517df816c9ad7e (127.13.162.3:38467): tablet a9f50c0a300347afaa4b0c3324017cfc (table test-table [id=7ed085a63362424c80f76a5630a6b9ba]) successfully deleted
I20260924 10:06:16.491851 32130 ts_tablet_manager.cc:1918] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:16.492136 32130 log.cc:1196] T a9f50c0a300347afaa4b0c3324017cfc P ffff7ebd350841efaeb03d588edfcd6e: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/a9f50c0a300347afaa4b0c3324017cfc
I20260924 10:06:16.492640 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31699
W20260924 10:06:16.507484 31626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:38183 (error 108)
I20260924 10:06:16.508203 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31830
I20260924 10:06:16.509449 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.509030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58292)
0924 10:06:16.509206 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.509309 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:06:16.509665 31626 catalog_manager.cc:4584] TS ffff7ebd350841efaeb03d588edfcd6e (127.13.162.1:38183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9f50c0a300347afaa4b0c3324017cfc: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
I20260924 10:06:16.515168 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.508993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.13.162.2:32951)
0924 10:06:16.509172 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.509178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.509199 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:06:16.509401 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.509406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.509505 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.509597 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:16.510106 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.511095 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.512362 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.512773 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.512868 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.513757 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:16.513765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:16.513778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:16.514459 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:16.514513 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:16.514536 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:16.514629 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.514744 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.514838 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:06:16.515868 31952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:16.563547 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.562974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58300)
0924 10:06:16.563216 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.563340 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:06:16.564666 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.564343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58308)
0924 10:06:16.564472 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.564543 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:06:16.605163 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31830
I20260924 10:06:16.618904 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31960
I20260924 10:06:16.625429 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.619614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.13.162.3:38467)
0924 10:06:16.619783 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:16.619788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:16.619807 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:06:16.619897 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:16.619900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:16.619997 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:16.620087 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:16.620471 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.621198 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.622449 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:16.622845 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:16.622950 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:16.624026 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:16.624035 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:16.624051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:16.624651 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:16.624700 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:16.624722 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:16.624870 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:16.624990 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:06:16.625106 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:06:16.625864 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.625352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58314)
0924 10:06:16.625576 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.625672 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:06:16.626124 32082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:16.627079 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.626626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58316)
0924 10:06:16.626796 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.626920 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:06:16.707756 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.707252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58320)
0924 10:06:16.707514 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.707608 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 10:06:16.708801 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:16.708399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:38183 (local address 127.0.0.1:58334)
0924 10:06:16.708597 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:16.708675 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 10:06:16.728989 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31960
I20260924 10:06:16.742614 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 31609
I20260924 10:06:16.743733 31670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:16.850353 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 31609
2026-09-24T10:06:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2663 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:16 dist-test-slave-jvsj krb5kdc[32140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:16 dist-test-slave-jvsj krb5kdc[32140](info): set up 2 sockets
Sep 24 10:06:16 dist-test-slave-jvsj krb5kdc[32140](info): commencing operation
krb5kdc: starting...
W20260924 10:06:17.862181 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:18 dist-test-slave-jvsj krb5kdc[32140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:18.091593 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44311
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:18.415336 32153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:18.415657 32153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:18.415789 32153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:18.423151 32153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:18.423260 32153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:18.423338 32153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:18.423420 32153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:18.423480 32153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:18.433344 32153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:18.434746 32153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:18.436108 32153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:18.447672 32158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:18.447642 32161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:18.447640 32159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:18.448408 32153 server_base.cc:1034] running on GCE node
I20260924 10:06:18.449132 32153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:18.450541 32153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:18.451725 32153 hybrid_clock.cc:648] HybridClock initialized: now 1790244378451671 us; error 47 us; skew 500 ppm
I20260924 10:06:18.452050 32153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:18 dist-test-slave-jvsj krb5kdc[32140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244378, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:18.455020 32153 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:06:18.456374 32153 webserver.cc:460] Webserver started at http://127.13.162.62:35289/ using document root <none> and password file <none>
I20260924 10:06:18.456981 32153 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:18.457075 32153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:18.457307 32153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:18.460131 32153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "8a3907d05bff4840b6da9af798c84799"
format_stamp: "Formatted at 2026-09-24 10:06:18 on dist-test-slave-jvsj"
server_key: "d63171907e4206519da598f194b3dac2"
server_key_iv: "25127da05f641ee96235e166eedc4d05"
server_key_version: "encryptionkey@0"
I20260924 10:06:18.460891 32153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "8a3907d05bff4840b6da9af798c84799"
format_stamp: "Formatted at 2026-09-24 10:06:18 on dist-test-slave-jvsj"
server_key: "d63171907e4206519da598f194b3dac2"
server_key_iv: "25127da05f641ee96235e166eedc4d05"
server_key_version: "encryptionkey@0"
I20260924 10:06:18.468058 32153 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260924 10:06:18.472535 32168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:18.474413 32153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:06:18.474627 32153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "8a3907d05bff4840b6da9af798c84799"
format_stamp: "Formatted at 2026-09-24 10:06:18 on dist-test-slave-jvsj"
server_key: "d63171907e4206519da598f194b3dac2"
server_key_iv: "25127da05f641ee96235e166eedc4d05"
server_key_version: "encryptionkey@0"
I20260924 10:06:18.474854 32153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:18.506500 32153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:18.510608 32153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:18.510888 32153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:18.527904 32153 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:44311
I20260924 10:06:18.527915 32219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:44311 every 8 connection(s)
I20260924 10:06:18.529234 32153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:18.532904 32220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:18.533931 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 32153
I20260924 10:06:18.534113 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:18.534619 13960 external_mini_cluster.cc:1428] Setting key fc1b5bba54682c7bb78fb2dbbe99f0e8
I20260924 10:06:18.546428 32220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799: Bootstrap starting.
Sep 24 10:06:18 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:18.550304 32220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:18.551545 32220 log.cc:824] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:18.554801 32220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799: No bootstrap required, opened a new log
I20260924 10:06:18.557899 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:18.536547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.13.162.62:44311)
0924 10:06:18.537275 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:18.537286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:18.537328 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:06:18.538976 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:18.538998 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:18.539418 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:18.539736 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:18.541397 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:18.542744 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:18.544639 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:18.545315 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:18.545444 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:18.550275 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:18.550283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:18.550301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:18.550385 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:18.553713 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:18.554603 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:18.554609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:18.554615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:18.554782 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:18.555216 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:18.555220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:18.555222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:18.555754 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:18.555963 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:18.556467 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 10:06:18.556800 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":321,"threads_started":1}
I20260924 10:06:18.563724 32220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3907d05bff4840b6da9af798c84799" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44311 } }
I20260924 10:06:18.564061 32220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:18.564153 32220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3907d05bff4840b6da9af798c84799, State: Initialized, Role: FOLLOWER
I20260924 10:06:18.564714 32220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3907d05bff4840b6da9af798c84799" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44311 } }
I20260924 10:06:18.564932 32220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:18.565053 32220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:18.565203 32220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:18.567329 32220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3907d05bff4840b6da9af798c84799" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44311 } }
I20260924 10:06:18.567790 32220 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a3907d05bff4840b6da9af798c84799; no voters:
I20260924 10:06:18.568308 32220 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:18.568435 32225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:18.569041 32225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3907d05bff4840b6da9af798c84799, State: Running, Role: LEADER
I20260924 10:06:18.569538 32225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3907d05bff4840b6da9af798c84799" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44311 } }
I20260924 10:06:18.570398 32220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:18.572321 32226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a3907d05bff4840b6da9af798c84799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3907d05bff4840b6da9af798c84799" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44311 } } }
I20260924 10:06:18.572733 32226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:18.573149 32227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3907d05bff4840b6da9af798c84799. Latest consensus state: current_term: 1 leader_uuid: "8a3907d05bff4840b6da9af798c84799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3907d05bff4840b6da9af798c84799" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44311 } } }
I20260924 10:06:18.573421 32227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:18.575186 32232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:18.580044 32232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:18.589845 32232 catalog_manager.cc:1347] Generated new cluster ID: cdc3e6fb944c46189a7f08d503961bd5
I20260924 10:06:18.589972 32232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:18.605108 32232 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:18.606498 32232 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:18.613994 32232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799: Generated new TSK 0
I20260924 10:06:18.615106 32232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:06:18.675690 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44311
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:18.999492 32246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:18.999776 32246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:18.999902 32246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:19.007220 32246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:19.007330 32246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:19.007491 32246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:06:19.017428 32246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:19.018782 32246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:19.020085 32246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:19.032904 32251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:19.032938 32252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:19.032967 32254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:19.033747 32246 server_base.cc:1034] running on GCE node
I20260924 10:06:19.034390 32246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:19.035341 32246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:19.036543 32246 hybrid_clock.cc:648] HybridClock initialized: now 1790244379036484 us; error 49 us; skew 500 ppm
I20260924 10:06:19.036854 32246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:19 dist-test-slave-jvsj krb5kdc[32140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244379, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:19.039928 32246 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:06:19.041399 32246 webserver.cc:460] Webserver started at http://127.13.162.1:36389/ using document root <none> and password file <none>
I20260924 10:06:19.042083 32246 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:19.042197 32246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:19.042526 32246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:19.045372 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3"
format_stamp: "Formatted at 2026-09-24 10:06:19 on dist-test-slave-jvsj"
server_key: "a22b548c55df4c46e888b7d6f2d0fe3a"
server_key_iv: "bc0369a97aac9a18cfefdccff615fe2b"
server_key_version: "encryptionkey@0"
I20260924 10:06:19.046154 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3"
format_stamp: "Formatted at 2026-09-24 10:06:19 on dist-test-slave-jvsj"
server_key: "a22b548c55df4c46e888b7d6f2d0fe3a"
server_key_iv: "bc0369a97aac9a18cfefdccff615fe2b"
server_key_version: "encryptionkey@0"
I20260924 10:06:19.051591 32246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:06:19.055294 32261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:19.056655 32246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:06:19.056821 32246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3"
format_stamp: "Formatted at 2026-09-24 10:06:19 on dist-test-slave-jvsj"
server_key: "a22b548c55df4c46e888b7d6f2d0fe3a"
server_key_iv: "bc0369a97aac9a18cfefdccff615fe2b"
server_key_version: "encryptionkey@0"
I20260924 10:06:19.057027 32246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:19.083165 32246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:19.087505 32246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:19.087781 32246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:19.088721 32246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:19.090080 32246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:19.090155 32246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:19.090268 32246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:19.090337 32246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:19.119681 32246 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:42295
I20260924 10:06:19.119736 32373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:42295 every 8 connection(s)
I20260924 10:06:19.120920 32246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:06:19.127638 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 32246
I20260924 10:06:19.127799 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:06:19.128227 13960 external_mini_cluster.cc:1428] Setting key 88017ea67ff5666cc2a29dfcd8fad410
Sep 24 10:06:19 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244379, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:19.141721 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:19.123375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:45935 (local address 127.13.162.62:44311)
0924 10:06:19.123781 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:19.123786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:19.124748 (+ 962us) server_negotiation.cc:407] Connection header received
0924 10:06:19.126591 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:19.126596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:19.126674 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:19.126767 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:19.128278 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.129129 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.131161 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.131564 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.131662 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:19.135410 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:19.135417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:19.135421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:19.135463 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:19.138557 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:19.139486 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:19.139490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:19.139493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:19.139649 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:19.140114 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:19.140117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:19.140119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:19.140372 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:19.140477 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:19.141292 (+ 815us) server_negotiation.cc:299] Negotiation successful
0924 10:06:19.141460 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":199,"threads_started":1}
I20260924 10:06:19.143102 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:19.123931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44311 (local address 127.13.162.1:45935)
0924 10:06:19.124600 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:19.124636 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:19.126307 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:19.127003 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:19.127011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:19.127505 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:19.128118 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:19.128136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.129293 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.129297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:19.131024 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:19.131038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.131782 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.131789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:19.132058 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:19.132723 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:19.132749 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:19.135174 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:19.138716 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:19.138725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:19.138745 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:19.139354 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:19.139755 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:19.139759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:19.139763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:19.140006 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:19.140480 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:19.140489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:19.140903 (+ 414us) client_negotiation.cc:769] Sending connection context
0924 10:06:19.141200 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 10:06:19.141722 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260924 10:06:19.144640 32374 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44311
I20260924 10:06:19.144928 32374 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:19.145655 32374 heartbeater.cc:511] Master 127.13.162.62:44311 requested a full tablet report, sending...
I20260924 10:06:19.147763 32185 ts_manager.cc:194] Registered new tserver with Master: ee194f4e20cc4aff8f4d2836d9ee56d3 (127.13.162.1:42295)
I20260924 10:06:19.149217 32185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:45935
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:06:19.179746 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44311
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:19.496801 32380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:19.497088 32380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:19.497212 32380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:19.504464 32380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:19.504604 32380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:19.504756 32380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:06:19.514627 32380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:19.516006 32380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:19.517307 32380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:19.529764 32386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:19.530035 32388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:19.530224 32385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:19.530983 32380 server_base.cc:1034] running on GCE node
I20260924 10:06:19.531566 32380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:19.532490 32380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:19.533694 32380 hybrid_clock.cc:648] HybridClock initialized: now 1790244379533633 us; error 50 us; skew 500 ppm
I20260924 10:06:19.533995 32380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:19 dist-test-slave-jvsj krb5kdc[32140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244379, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:19.537146 32380 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:06:19.538698 32380 webserver.cc:460] Webserver started at http://127.13.162.2:38497/ using document root <none> and password file <none>
I20260924 10:06:19.539376 32380 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:19.539490 32380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:19.539812 32380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:19.542783 32380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "42b46d6892124e2280e1b51e2ef581c2"
format_stamp: "Formatted at 2026-09-24 10:06:19 on dist-test-slave-jvsj"
server_key: "3e9de89d0c3fa762e390948f5b4ac56e"
server_key_iv: "c36634adc1003a71ef278d87990d8cd2"
server_key_version: "encryptionkey@0"
I20260924 10:06:19.543531 32380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "42b46d6892124e2280e1b51e2ef581c2"
format_stamp: "Formatted at 2026-09-24 10:06:19 on dist-test-slave-jvsj"
server_key: "3e9de89d0c3fa762e390948f5b4ac56e"
server_key_iv: "c36634adc1003a71ef278d87990d8cd2"
server_key_version: "encryptionkey@0"
I20260924 10:06:19.548914 32380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:19.552702 32395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:19.554136 32380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:06:19.554302 32380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "42b46d6892124e2280e1b51e2ef581c2"
format_stamp: "Formatted at 2026-09-24 10:06:19 on dist-test-slave-jvsj"
server_key: "3e9de89d0c3fa762e390948f5b4ac56e"
server_key_iv: "c36634adc1003a71ef278d87990d8cd2"
server_key_version: "encryptionkey@0"
I20260924 10:06:19.554545 32380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:19.588798 32380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:19.593339 32380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:19.593626 32380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:19.594660 32380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:19.596050 32380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:19.596122 32380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:19.596241 32380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:19.596297 32380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:19.624624 32380 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:34975
I20260924 10:06:19.624672 32507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:34975 every 8 connection(s)
I20260924 10:06:19.625871 32380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:06:19.631872 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 32380
I20260924 10:06:19.632077 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:06:19.632478 13960 external_mini_cluster.cc:1428] Setting key 14b7c2b726158d48c9babea57160ef44
Sep 24 10:06:19 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244379, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:19.646733 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:19.628364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:57159 (local address 127.13.162.62:44311)
0924 10:06:19.628533 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:19.628537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:19.629705 (+ 1168us) server_negotiation.cc:407] Connection header received
0924 10:06:19.631555 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:19.631561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:19.631654 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:19.631785 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:19.633370 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.634416 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.636647 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.637103 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.637196 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:19.641094 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:19.641099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:19.641102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:19.641135 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:19.643840 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:19.644602 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:19.644609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:19.644613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:19.644762 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:19.645106 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:19.645110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:19.645114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:19.645425 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:19.645553 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:19.646191 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 10:06:19.646428 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:06:19.647670 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:19.628839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44311 (local address 127.13.162.2:57159)
0924 10:06:19.629510 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:19.629571 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:19.631257 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:19.631973 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:19.631982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:19.632584 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:19.633184 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:19.633209 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.634579 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.634584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:19.636506 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:19.636527 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:19.637316 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:19.637322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:19.637611 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:19.638270 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:19.638302 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:19.640848 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:19.644004 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:19.644034 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:19.644054 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:19.644482 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:19.644851 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:19.644854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:19.644856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:19.645022 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:19.645524 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:19.645529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:19.645856 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 10:06:19.646086 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 10:06:19.646504 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":262,"threads_started":1}
I20260924 10:06:19.649137 32508 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44311
I20260924 10:06:19.649459 32508 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:19.650305 32508 heartbeater.cc:511] Master 127.13.162.62:44311 requested a full tablet report, sending...
I20260924 10:06:19.651907 32185 ts_manager.cc:194] Registered new tserver with Master: 42b46d6892124e2280e1b51e2ef581c2 (127.13.162.2:34975)
I20260924 10:06:19.652714 32185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:57159
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:06:19.685292 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44311
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:20.000761 32513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:20.001055 32513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:20.001186 32513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:20.008395 32513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:20.008534 32513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:20.008690 32513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:06:20.018442 32513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:44311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:20.019739 32513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:20.021046 32513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:20.033962 32518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:20.033998 32519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:20.034018 32521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:20.039831 32513 server_base.cc:1034] running on GCE node
I20260924 10:06:20.040531 32513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:20.041474 32513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:20.042665 32513 hybrid_clock.cc:648] HybridClock initialized: now 1790244380042599 us; error 55 us; skew 500 ppm
I20260924 10:06:20.042984 32513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:20 dist-test-slave-jvsj krb5kdc[32140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244380, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:20.046066 32513 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:06:20.047744 32513 webserver.cc:460] Webserver started at http://127.13.162.3:36617/ using document root <none> and password file <none>
I20260924 10:06:20.048504 32513 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:20.048633 32513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:20.048951 32513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:20.051891 32513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1"
format_stamp: "Formatted at 2026-09-24 10:06:20 on dist-test-slave-jvsj"
server_key: "5d85e30361b54ca3bc2efaf4dfbd7fac"
server_key_iv: "9a56fd2a93af370e43828a613dd49ef4"
server_key_version: "encryptionkey@0"
I20260924 10:06:20.052649 32513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1"
format_stamp: "Formatted at 2026-09-24 10:06:20 on dist-test-slave-jvsj"
server_key: "5d85e30361b54ca3bc2efaf4dfbd7fac"
server_key_iv: "9a56fd2a93af370e43828a613dd49ef4"
server_key_version: "encryptionkey@0"
I20260924 10:06:20.057983 32513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:06:20.061659 32528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:20.063110 32513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:06:20.063257 32513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1"
format_stamp: "Formatted at 2026-09-24 10:06:20 on dist-test-slave-jvsj"
server_key: "5d85e30361b54ca3bc2efaf4dfbd7fac"
server_key_iv: "9a56fd2a93af370e43828a613dd49ef4"
server_key_version: "encryptionkey@0"
I20260924 10:06:20.063452 32513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:20.089047 32513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:20.093212 32513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:20.093493 32513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:20.094518 32513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:20.095883 32513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:20.095958 32513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:20.096074 32513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:20.096124 32513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:20.126609 32513 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:39307
I20260924 10:06:20.126693 32640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:39307 every 8 connection(s)
I20260924 10:06:20.127823 32513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:06:20.136709 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 32513
I20260924 10:06:20.136889 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:06:20.137319 13960 external_mini_cluster.cc:1428] Setting key 77afc9294b9f66899604d0def5975586
Sep 24 10:06:20 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244380, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:20.146970 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.130188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:46037 (local address 127.13.162.62:44311)
0924 10:06:20.130426 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.130432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.131579 (+ 1147us) server_negotiation.cc:407] Connection header received
0924 10:06:20.133367 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.133372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.133444 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.133540 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:20.135042 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.135795 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.137439 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.137842 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.137957 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.141494 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:20.141499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:20.141502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:20.141537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:20.143941 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.144669 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.144673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.144675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.144804 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.145217 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.145221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.145223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.145498 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:20.145635 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.146390 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.146627 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:06:20.147945 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.130697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:44311 (local address 127.13.162.3:46037)
0924 10:06:20.131370 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:20.131414 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:20.133077 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:20.133742 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:20.133750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:20.134243 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:20.134865 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.134883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.135938 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.135942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.137298 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.137311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.138058 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.138065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.138297 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.139098 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:20.139123 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:20.141296 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:20.144074 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:20.144084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:20.144112 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:20.144545 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:20.144907 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:20.144912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:20.144916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:20.145103 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:20.145632 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:20.145640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:20.145983 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 10:06:20.146223 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 10:06:20.146848 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":261,"threads_started":1}
I20260924 10:06:20.149552 32641 heartbeater.cc:347] Connected to a master server at 127.13.162.62:44311
I20260924 10:06:20.149837 32641 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:20.150595 32641 heartbeater.cc:511] Master 127.13.162.62:44311 requested a full tablet report, sending...
I20260924 10:06:20.152144 32185 ts_manager.cc:194] Registered new tserver with Master: b6e38b91a0f549cb8a20b01e4a9c66e1 (127.13.162.3:39307)
I20260924 10:06:20.152359 32374 heartbeater.cc:503] Master 127.13.162.62:44311 was elected leader, sending a full tablet report...
I20260924 10:06:20.152810 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:06:20.153227 32185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:46037
I20260924 10:06:20.173380 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.157226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.13.162.62:44311)
0924 10:06:20.157384 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.157388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.157825 (+ 437us) server_negotiation.cc:407] Connection header received
0924 10:06:20.157976 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.157980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.158063 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.158142 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:20.159515 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.160227 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.161498 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.161919 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.162012 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.168190 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:20.168198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:20.168204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:20.168248 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:20.170833 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.171586 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.171591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.171593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.171772 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.172221 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.172224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.172226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.172529 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:20.172728 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.173016 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.173159 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:06:20.177372 32185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:06:20.184597 32185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:06:20.207341 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.196209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:42295 (local address 127.0.0.1:47456)
0924 10:06:20.197459 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:20.197509 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:20.197719 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:20.198847 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:20.198855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:20.198888 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:20.199397 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.199411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.201483 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.201490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.204005 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.204023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.206303 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.206310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.206537 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.206601 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 10:06:20.206736 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:06:20.206931 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"threads_started":1}
I20260924 10:06:20.208246 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.196451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.13.162.1:42295)
0924 10:06:20.197075 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.197085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.197918 (+ 833us) server_negotiation.cc:407] Connection header received
0924 10:06:20.198038 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.198047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.198341 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.198546 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:20.199624 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.201306 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.204210 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.205776 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.205912 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.206152 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.207738 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.207922 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":291,"threads_started":1}
I20260924 10:06:20.213049 32309 tablet_service.cc:1467] Processing CreateTablet for tablet f7652134115a441cafcc9a0213f44715 (DEFAULT_TABLE table=test-table [id=b4b7dba062ef413299cce92a17d3013a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:20.214612 32309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7652134115a441cafcc9a0213f44715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:20.216780 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.197874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:34975 (local address 127.0.0.1:57566)
0924 10:06:20.208762 (+ 10888us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:20.208780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:20.208893 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:20.210780 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:20.210784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:20.210809 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:20.211177 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.211189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.212821 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.212825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.214516 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.214529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.215930 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.215935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.216098 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.216130 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:06:20.216212 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:06:20.216359 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10740,"mutex_wait_us":1,"thread_start_us":11109,"threads_started":1}
I20260924 10:06:20.216863 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.198170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.13.162.2:34975)
0924 10:06:20.210189 (+ 12019us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.210197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.210224 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:06:20.210333 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.210342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.210589 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.210770 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:20.211360 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.212705 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.214711 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.215738 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.215857 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.216061 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.216384 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.216560 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11861,"thread_start_us":11761,"threads_started":1}
I20260924 10:06:20.216852 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.196544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:39307 (local address 127.0.0.1:59234)
0924 10:06:20.208593 (+ 12049us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:20.208619 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:20.208724 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:20.210252 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:20.210257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:20.210283 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:20.210455 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.210465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.212517 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.212522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.214408 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.214423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.216160 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.216165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.216348 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.216376 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:06:20.216459 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:06:20.216584 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11886,"mutex_wait_us":44,"thread_start_us":537,"threads_started":1}
I20260924 10:06:20.217615 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.197188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.13.162.3:39307)
0924 10:06:20.209589 (+ 12401us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.209599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.209635 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:06:20.209732 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.209741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.210022 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.210625 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:20.210923 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.212370 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.214591 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.215940 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.216071 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.216274 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.216982 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.217355 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12215,"thread_start_us":12145,"threads_started":1}
I20260924 10:06:20.220597 32443 tablet_service.cc:1467] Processing CreateTablet for tablet f7652134115a441cafcc9a0213f44715 (DEFAULT_TABLE table=test-table [id=b4b7dba062ef413299cce92a17d3013a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:20.221277 32576 tablet_service.cc:1467] Processing CreateTablet for tablet f7652134115a441cafcc9a0213f44715 (DEFAULT_TABLE table=test-table [id=b4b7dba062ef413299cce92a17d3013a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:20.222023 32443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7652134115a441cafcc9a0213f44715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:20.222707 32576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7652134115a441cafcc9a0213f44715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:20.230113 32659 tablet_bootstrap.cc:492] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Bootstrap starting.
I20260924 10:06:20.233827 32660 tablet_bootstrap.cc:492] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Bootstrap starting.
I20260924 10:06:20.234015 32659 tablet_bootstrap.cc:654] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:20.235298 32659 log.cc:824] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:20.235957 32661 tablet_bootstrap.cc:492] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Bootstrap starting.
I20260924 10:06:20.237674 32660 tablet_bootstrap.cc:654] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:20.239396 32661 tablet_bootstrap.cc:654] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:20.239395 32660 log.cc:824] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:20.240700 32661 log.cc:824] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:20.243737 32660 tablet_bootstrap.cc:492] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: No bootstrap required, opened a new log
I20260924 10:06:20.243875 32659 tablet_bootstrap.cc:492] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: No bootstrap required, opened a new log
I20260924 10:06:20.243875 32661 tablet_bootstrap.cc:492] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: No bootstrap required, opened a new log
I20260924 10:06:20.244019 32660 ts_tablet_manager.cc:1397] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 10:06:20.244156 32661 ts_tablet_manager.cc:1397] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260924 10:06:20.244146 32659 ts_tablet_manager.cc:1397] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260924 10:06:20.251706 32660 raft_consensus.cc:348] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.252084 32660 raft_consensus.cc:374] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:20.252173 32660 raft_consensus.cc:729] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b46d6892124e2280e1b51e2ef581c2, State: Initialized, Role: FOLLOWER
I20260924 10:06:20.252763 32660 consensus_queue.cc:260] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.253767 32508 heartbeater.cc:503] Master 127.13.162.62:44311 was elected leader, sending a full tablet report...
I20260924 10:06:20.254570 32660 ts_tablet_manager.cc:1428] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260924 10:06:20.254493 32661 raft_consensus.cc:348] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.254537 32659 raft_consensus.cc:348] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.254925 32661 raft_consensus.cc:374] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:20.254925 32659 raft_consensus.cc:374] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:20.255015 32659 raft_consensus.cc:729] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee194f4e20cc4aff8f4d2836d9ee56d3, State: Initialized, Role: FOLLOWER
I20260924 10:06:20.255016 32661 raft_consensus.cc:729] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e38b91a0f549cb8a20b01e4a9c66e1, State: Initialized, Role: FOLLOWER
I20260924 10:06:20.255684 32661 consensus_queue.cc:260] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.255721 32659 consensus_queue.cc:260] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.257508 32659 ts_tablet_manager.cc:1428] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 10:06:20.259379 32641 heartbeater.cc:503] Master 127.13.162.62:44311 was elected leader, sending a full tablet report...
I20260924 10:06:20.259979 32661 ts_tablet_manager.cc:1428] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260924 10:06:20.319502 32665 raft_consensus.cc:482] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:06:20.319837 32665 raft_consensus.cc:504] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.321516 32665 leader_election.cc:288] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee194f4e20cc4aff8f4d2836d9ee56d3 (127.13.162.1:42295), b6e38b91a0f549cb8a20b01e4a9c66e1 (127.13.162.3:39307)
I20260924 10:06:20.328402 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.322219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:41087 (local address 127.13.162.3:39307)
0924 10:06:20.322465 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.322471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.323386 (+ 915us) server_negotiation.cc:407] Connection header received
0924 10:06:20.323527 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.323537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.323596 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.323668 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:20.324297 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.325022 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.326798 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.327521 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.327675 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.327747 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.328042 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.328165 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:06:20.328408 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.322219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:39117 (local address 127.13.162.1:42295)
0924 10:06:20.322638 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.322644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.323292 (+ 648us) server_negotiation.cc:407] Connection header received
0924 10:06:20.323477 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.323481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.323538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.323608 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:20.324149 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.324831 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.326812 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.327580 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.327718 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.327787 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.328046 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.328180 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260924 10:06:20.328732 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.321994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:39307 (local address 127.13.162.2:41087)
0924 10:06:20.323265 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:20.323284 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:20.323398 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:20.323702 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:20.323706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:20.323791 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:20.324147 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.324156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.325157 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.325161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.326651 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.326665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.327745 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.327753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.327899 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.327922 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:06:20.328000 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:06:20.328158 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":6,"spinlock_wait_cycles":26368,"thread_start_us":828,"threads_started":1}
I20260924 10:06:20.328770 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.321802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:42295 (local address 127.13.162.2:39117)
0924 10:06:20.323188 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:20.323207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:20.323348 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:20.323638 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:20.323643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:20.323665 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:20.324015 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.324023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.324951 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.324956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.326672 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:20.326684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.327779 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.327785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:20.327913 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.327934 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:06:20.328010 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:06:20.328180 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":135,"thread_start_us":1162,"threads_started":1}
I20260924 10:06:20.329728 32596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7652134115a441cafcc9a0213f44715" candidate_uuid: "42b46d6892124e2280e1b51e2ef581c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" is_pre_election: true
I20260924 10:06:20.329735 32329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7652134115a441cafcc9a0213f44715" candidate_uuid: "42b46d6892124e2280e1b51e2ef581c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" is_pre_election: true
I20260924 10:06:20.330185 32596 raft_consensus.cc:2454] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b46d6892124e2280e1b51e2ef581c2 in term 0.
I20260924 10:06:20.330184 32329 raft_consensus.cc:2454] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b46d6892124e2280e1b51e2ef581c2 in term 0.
I20260924 10:06:20.330809 32399 leader_election.cc:302] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b46d6892124e2280e1b51e2ef581c2, b6e38b91a0f549cb8a20b01e4a9c66e1; no voters:
I20260924 10:06:20.331166 32665 raft_consensus.cc:2781] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:06:20.331282 32665 raft_consensus.cc:482] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:06:20.331374 32665 raft_consensus.cc:3037] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:20.333470 32665 raft_consensus.cc:504] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.334087 32665 leader_election.cc:288] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [CANDIDATE]: Term 1 election: Requested vote from peers ee194f4e20cc4aff8f4d2836d9ee56d3 (127.13.162.1:42295), b6e38b91a0f549cb8a20b01e4a9c66e1 (127.13.162.3:39307)
I20260924 10:06:20.334512 32329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7652134115a441cafcc9a0213f44715" candidate_uuid: "42b46d6892124e2280e1b51e2ef581c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3"
I20260924 10:06:20.334602 32596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7652134115a441cafcc9a0213f44715" candidate_uuid: "42b46d6892124e2280e1b51e2ef581c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1"
I20260924 10:06:20.334758 32329 raft_consensus.cc:3037] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:20.334779 32596 raft_consensus.cc:3037] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:20.337847 32329 raft_consensus.cc:2454] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b46d6892124e2280e1b51e2ef581c2 in term 1.
I20260924 10:06:20.337862 32596 raft_consensus.cc:2454] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b46d6892124e2280e1b51e2ef581c2 in term 1.
I20260924 10:06:20.338258 32398 leader_election.cc:302] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b46d6892124e2280e1b51e2ef581c2, ee194f4e20cc4aff8f4d2836d9ee56d3; no voters:
I20260924 10:06:20.338614 32665 raft_consensus.cc:2781] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:20.339036 32665 raft_consensus.cc:686] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 1 LEADER]: Becoming Leader. State: Replica: 42b46d6892124e2280e1b51e2ef581c2, State: Running, Role: LEADER
I20260924 10:06:20.339483 32665 consensus_queue.cc:237] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } }
I20260924 10:06:20.344336 32183 catalog_manager.cc:5495] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42b46d6892124e2280e1b51e2ef581c2 (127.13.162.2). New cstate: current_term: 1 leader_uuid: "42b46d6892124e2280e1b51e2ef581c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b46d6892124e2280e1b51e2ef581c2" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260924 10:06:20.364499 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.359327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.13.162.2:34975)
0924 10:06:20.359536 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.359540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.359570 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:06:20.359749 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.359753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.359840 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.359960 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:06:20.360304 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.361061 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.362450 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.362834 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.362925 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.363401 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:06:20.363900 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:06:20.364020 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.364169 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.364216 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:06:20.372889 32596 raft_consensus.cc:1261] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 1 FOLLOWER]: Refusing update from remote peer 42b46d6892124e2280e1b51e2ef581c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:20.372889 32329 raft_consensus.cc:1261] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 1 FOLLOWER]: Refusing update from remote peer 42b46d6892124e2280e1b51e2ef581c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 10:06:20.373158 32375 tablet.cc:2369] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:20.373950 32665 consensus_queue.cc:1035] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6e38b91a0f549cb8a20b01e4a9c66e1" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:20.374284 32670 consensus_queue.cc:1035] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee194f4e20cc4aff8f4d2836d9ee56d3" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:06:20.377645 32509 tablet.cc:2369] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:20.379319 32642 tablet.cc:2369] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:20.390928 32674 mvcc.cc:205] Tried to move back new op lower bound from 7332840981997166592 to 7332840981874597888. Current Snapshot: MvccSnapshot[applied={T|T < 7332840981997166592}]
I20260924 10:06:20.399065 32183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:20.399370 32183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:20.403314 32183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a3907d05bff4840b6da9af798c84799: Sending DeleteTablet for 3 replicas of tablet f7652134115a441cafcc9a0213f44715
I20260924 10:06:20.404505 32576 tablet_service.cc:1514] Processing DeleteTablet for tablet f7652134115a441cafcc9a0213f44715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:20 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:59234
I20260924 10:06:20.404691 32309 tablet_service.cc:1514] Processing DeleteTablet for tablet f7652134115a441cafcc9a0213f44715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:20 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:47456
I20260924 10:06:20.404733 32443 tablet_service.cc:1514] Processing DeleteTablet for tablet f7652134115a441cafcc9a0213f44715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:20 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:57566
I20260924 10:06:20.408449 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 32246
I20260924 10:06:20.415745 32684 tablet_replica.cc:330] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: stopping tablet replica
I20260924 10:06:20.416394 32684 raft_consensus.cc:2229] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:06:20.417166 32684 raft_consensus.cc:2258] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:20.417902 32682 tablet_replica.cc:330] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: stopping tablet replica
I20260924 10:06:20.418249 32682 raft_consensus.cc:2229] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:20.418766 32682 raft_consensus.cc:2258] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:20.420399 32684 ts_tablet_manager.cc:1905] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:20.420820 32682 ts_tablet_manager.cc:1905] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:20.427620 32682 ts_tablet_manager.cc:1918] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:20.427634 32684 ts_tablet_manager.cc:1918] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:20.427803 32682 log.cc:1196] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/f7652134115a441cafcc9a0213f44715
I20260924 10:06:20.427997 32684 log.cc:1196] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/f7652134115a441cafcc9a0213f44715
I20260924 10:06:20.428336 32682 ts_tablet_manager.cc:1939] T f7652134115a441cafcc9a0213f44715 P b6e38b91a0f549cb8a20b01e4a9c66e1: Deleting consensus metadata
I20260924 10:06:20.428679 32684 ts_tablet_manager.cc:1939] T f7652134115a441cafcc9a0213f44715 P 42b46d6892124e2280e1b51e2ef581c2: Deleting consensus metadata
I20260924 10:06:20.430621 32172 catalog_manager.cc:4841] TS b6e38b91a0f549cb8a20b01e4a9c66e1 (127.13.162.3:39307): tablet f7652134115a441cafcc9a0213f44715 (table test-table [id=b4b7dba062ef413299cce92a17d3013a]) successfully deleted
I20260924 10:06:20.431020 32172 catalog_manager.cc:4841] TS 42b46d6892124e2280e1b51e2ef581c2 (127.13.162.2:34975): tablet f7652134115a441cafcc9a0213f44715 (table test-table [id=b4b7dba062ef413299cce92a17d3013a]) successfully deleted
I20260924 10:06:20.434862 32683 tablet_replica.cc:330] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: stopping tablet replica
I20260924 10:06:20.435405 32683 raft_consensus.cc:2229] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:20.435925 32683 raft_consensus.cc:2258] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:20.438048 32683 ts_tablet_manager.cc:1905] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 10:06:20 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:06:20.444880 32683 ts_tablet_manager.cc:1918] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:20.445027 32683 log.cc:1196] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/f7652134115a441cafcc9a0213f44715
I20260924 10:06:20.445429 32683 ts_tablet_manager.cc:1939] T f7652134115a441cafcc9a0213f44715 P ee194f4e20cc4aff8f4d2836d9ee56d3: Deleting consensus metadata
I20260924 10:06:20.447716 32171 catalog_manager.cc:4841] TS ee194f4e20cc4aff8f4d2836d9ee56d3 (127.13.162.1:42295): tablet f7652134115a441cafcc9a0213f44715 (table test-table [id=b4b7dba062ef413299cce92a17d3013a]) successfully deleted
I20260924 10:06:20.453290 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.411832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.13.162.1:42295)
0924 10:06:20.412045 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.412051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.434788 (+ 22737us) server_negotiation.cc:407] Connection header received
0924 10:06:20.435068 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.435075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.435179 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.435267 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:20.437126 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.438165 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.440106 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.440780 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.440912 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.446142 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:20.446160 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:20.446177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:20.446219 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:20.450052 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.450948 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.450958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.450962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.451140 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.451574 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.451579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.451583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.452063 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:20.452360 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.452682 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.452906 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:06:20.453974 32369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:20.551870 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 32246
I20260924 10:06:20.566923 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 32380
Sep 24 10:06:20 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:06:20.584628 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.567634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.13.162.2:34975)
0924 10:06:20.567810 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.567815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.567894 (+ 79us) server_negotiation.cc:407] Connection header received
0924 10:06:20.568108 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.568115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.568190 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.568271 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:20.569984 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.570927 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.572774 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.573226 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.573328 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.578642 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:20.578660 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:20.578681 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:20.578728 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:20.581718 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.582590 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.582598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.582600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.582755 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.583142 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.583146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.583148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.583516 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:20.583825 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.584124 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.584302 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:06:20.585345 32503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:20.697932 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 32380
I20260924 10:06:20.713196 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 32513
Sep 24 10:06:20 dist-test-slave-jvsj krb5kdc[32140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:06:20.728338 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:20.713999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.13.162.3:39307)
0924 10:06:20.714161 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:20.714165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:20.714184 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:06:20.714288 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:20.714292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:20.714387 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:20.714467 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:20.715814 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.716561 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.717968 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:20.718399 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:20.718494 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:20.722936 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:20.722951 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:20.722968 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:20.723008 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:20.725422 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.726147 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.726155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.726158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.726386 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:20.726910 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:20.726914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:20.726916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:20.727351 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:20.727593 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:20.727884 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 10:06:20.728088 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:06:20.728960 32636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:20.817828 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 32513
I20260924 10:06:20.830987 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 32153
I20260924 10:06:20.832115 32215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:20.941828 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 32153
2026-09-24T10:06:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T10:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:20Z Disabled control of system clock
I20260924 10:06:21.012183 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:33625
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:06:21.348321 32693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:21.348623 32693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:21.348703 32693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:21.356118 32693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:21.356243 32693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:21.356281 32693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:21.356334 32693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:21.356383 32693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:21.365908 32693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:33625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:21.367187 32693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:21.368911 32693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:21.381064 32698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:21.380967 32701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:21.381099 32699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:21.382277 32693 server_base.cc:1034] running on GCE node
I20260924 10:06:21.383172 32693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:21.384843 32693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:21.386054 32693 hybrid_clock.cc:648] HybridClock initialized: now 1790244381385980 us; error 62 us; skew 500 ppm
I20260924 10:06:21.386476 32693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:21.388796 32693 webserver.cc:460] Webserver started at http://127.13.162.62:45515/ using document root <none> and password file <none>
I20260924 10:06:21.389505 32693 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:21.389598 32693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:21.389918 32693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:21.393013 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "b016ffa56ab44725a91c48c1827b5f9f"
format_stamp: "Formatted at 2026-09-24 10:06:21 on dist-test-slave-jvsj"
server_key: "cfad562296fc387508ffb0b6b51a0b27"
server_key_iv: "a096f93112763185e3a4d9c6acdd6bf1"
server_key_version: "encryptionkey@0"
I20260924 10:06:21.393844 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "b016ffa56ab44725a91c48c1827b5f9f"
format_stamp: "Formatted at 2026-09-24 10:06:21 on dist-test-slave-jvsj"
server_key: "cfad562296fc387508ffb0b6b51a0b27"
server_key_iv: "a096f93112763185e3a4d9c6acdd6bf1"
server_key_version: "encryptionkey@0"
I20260924 10:06:21.399224 32693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 10:06:21.402940 32707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:21.404476 32693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:21.404633 32693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "b016ffa56ab44725a91c48c1827b5f9f"
format_stamp: "Formatted at 2026-09-24 10:06:21 on dist-test-slave-jvsj"
server_key: "cfad562296fc387508ffb0b6b51a0b27"
server_key_iv: "a096f93112763185e3a4d9c6acdd6bf1"
server_key_version: "encryptionkey@0"
I20260924 10:06:21.404860 32693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:21.423192 32693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:21.424125 32693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:21.424399 32693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:21.442456 32693 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:33625
I20260924 10:06:21.442477 32758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:33625 every 8 connection(s)
I20260924 10:06:21.443804 32693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:21.444684 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 32693
I20260924 10:06:21.444854 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:21.445259 13960 external_mini_cluster.cc:1428] Setting key e5877c08bcd6125f22d59a9c9f30210d
I20260924 10:06:21.448107 32759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:21.461149 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:21.447188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.13.162.62:33625)
0924 10:06:21.448219 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:21.448235 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:21.448285 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:06:21.450585 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:21.450613 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:21.451225 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:21.451709 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:21.452353 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:21.454388 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:21.455861 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:21.456870 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:21.457067 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:21.458141 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:21.458152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:21.458171 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:21.458945 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:21.459061 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:21.459092 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:21.459213 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:21.459448 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:06:21.459808 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":313,"threads_started":1}
I20260924 10:06:21.462134 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f: Bootstrap starting.
I20260924 10:06:21.465430 32759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:21.466668 32759 log.cc:824] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:21.469349 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f: No bootstrap required, opened a new log
I20260924 10:06:21.477017 32759 raft_consensus.cc:348] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016ffa56ab44725a91c48c1827b5f9f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33625 } }
I20260924 10:06:21.477406 32759 raft_consensus.cc:374] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:21.477483 32759 raft_consensus.cc:729] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b016ffa56ab44725a91c48c1827b5f9f, State: Initialized, Role: FOLLOWER
I20260924 10:06:21.478056 32759 consensus_queue.cc:260] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016ffa56ab44725a91c48c1827b5f9f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33625 } }
I20260924 10:06:21.478259 32759 raft_consensus.cc:388] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:21.478389 32759 raft_consensus.cc:482] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:21.478528 32759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:21.480675 32759 raft_consensus.cc:504] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016ffa56ab44725a91c48c1827b5f9f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33625 } }
I20260924 10:06:21.481144 32759 leader_election.cc:302] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b016ffa56ab44725a91c48c1827b5f9f; no voters:
I20260924 10:06:21.481844 32759 leader_election.cc:288] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:21.481988 32764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:21.482815 32764 raft_consensus.cc:686] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [term 1 LEADER]: Becoming Leader. State: Replica: b016ffa56ab44725a91c48c1827b5f9f, State: Running, Role: LEADER
I20260924 10:06:21.483320 32764 consensus_queue.cc:237] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016ffa56ab44725a91c48c1827b5f9f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33625 } }
I20260924 10:06:21.483917 32759 sys_catalog.cc:564] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:21.485819 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b016ffa56ab44725a91c48c1827b5f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016ffa56ab44725a91c48c1827b5f9f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33625 } } }
I20260924 10:06:21.486310 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:21.487264 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b016ffa56ab44725a91c48c1827b5f9f. Latest consensus state: current_term: 1 leader_uuid: "b016ffa56ab44725a91c48c1827b5f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016ffa56ab44725a91c48c1827b5f9f" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 33625 } } }
I20260924 10:06:21.487548 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:21.490340 305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:21.495556 305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:21.503681 305 catalog_manager.cc:1347] Generated new cluster ID: 896abf6e715044079c14f514935fc116
I20260924 10:06:21.503816 305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:21.517279 305 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:21.518952 305 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:21.527839 305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f: Generated new TSK 0
I20260924 10:06:21.528772 305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:21.539824 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33625
--builtin_ntp_servers=127.13.162.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:06:21.861045 315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:21.861351 315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:21.861440 315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:21.868661 315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:21.868764 315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:21.868872 315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:06:21.878023 315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:21.879647 315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:21.881567 315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:21.894222 320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:21.894435 321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:21.894444 323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:21.895381 315 server_base.cc:1034] running on GCE node
I20260924 10:06:21.895952 315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:21.896889 315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:21.898099 315 hybrid_clock.cc:648] HybridClock initialized: now 1790244381898036 us; error 53 us; skew 500 ppm
I20260924 10:06:21.898465 315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:21.900899 315 webserver.cc:460] Webserver started at http://127.13.162.1:34809/ using document root <none> and password file <none>
I20260924 10:06:21.901563 315 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:21.901646 315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:21.901899 315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:21.904742 315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "7ce49464c72b4db3a0c114152448aa8b"
format_stamp: "Formatted at 2026-09-24 10:06:21 on dist-test-slave-jvsj"
server_key: "9a1dabf8ab6508ca7e9140916d52c030"
server_key_iv: "48472ebc14e4bbaf38ea8f361f6d53b4"
server_key_version: "encryptionkey@0"
I20260924 10:06:21.905493 315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "7ce49464c72b4db3a0c114152448aa8b"
format_stamp: "Formatted at 2026-09-24 10:06:21 on dist-test-slave-jvsj"
server_key: "9a1dabf8ab6508ca7e9140916d52c030"
server_key_iv: "48472ebc14e4bbaf38ea8f361f6d53b4"
server_key_version: "encryptionkey@0"
I20260924 10:06:21.910717 315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 10:06:21.914245 329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:21.915633 315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:21.915796 315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "7ce49464c72b4db3a0c114152448aa8b"
format_stamp: "Formatted at 2026-09-24 10:06:21 on dist-test-slave-jvsj"
server_key: "9a1dabf8ab6508ca7e9140916d52c030"
server_key_iv: "48472ebc14e4bbaf38ea8f361f6d53b4"
server_key_version: "encryptionkey@0"
I20260924 10:06:21.916016 315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:21.947530 315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:21.948408 315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:21.948700 315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:21.949681 315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:21.951224 315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:21.951325 315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:21.951431 315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:21.951505 315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:21.981027 315 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:35865
I20260924 10:06:21.981099 441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:35865 every 8 connection(s)
I20260924 10:06:21.982259 315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:06:21.992512 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 315
I20260924 10:06:21.992697 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:06:21.993096 13960 external_mini_cluster.cc:1428] Setting key b03781d2814f22e054bb6abb4778ea1a
I20260924 10:06:21.994397 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:21.984712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:47305 (local address 127.13.162.62:33625)
0924 10:06:21.985126 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:21.985132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:21.986407 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 10:06:21.988162 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:21.988167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:21.988262 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:21.988356 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:21.989414 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:21.990115 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:21.991690 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:21.992100 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:21.992230 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:21.993528 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:21.993533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:21.993536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:21.993553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:21.993610 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:21.993627 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:21.993717 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:21.993949 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:06:21.994027 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":204,"threads_started":1}
I20260924 10:06:21.995492 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:21.985388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33625 (local address 127.13.162.1:47305)
0924 10:06:21.986244 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:21.986289 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:21.987886 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:21.988600 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:21.988609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:21.988695 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:21.989235 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:21.989250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:21.990274 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:21.990278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:21.991547 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:21.991559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:21.992312 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:21.992319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:21.992548 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:21.993219 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:21.993223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:21.993273 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:21.993348 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:21.993733 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:21.993743 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:21.993750 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:06:21.993937 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 10:06:21.994294 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":241,"threads_started":1}
I20260924 10:06:21.996132 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33625
--builtin_ntp_servers=127.13.162.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:06:21.996886 442 heartbeater.cc:347] Connected to a master server at 127.13.162.62:33625
I20260924 10:06:21.997200 442 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:21.998040 442 heartbeater.cc:511] Master 127.13.162.62:33625 requested a full tablet report, sending...
I20260924 10:06:22.000046 32724 ts_manager.cc:194] Registered new tserver with Master: 7ce49464c72b4db3a0c114152448aa8b (127.13.162.1:35865)
I20260924 10:06:22.001515 32724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.1:47305
W20260924 10:06:22.312723 446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:22.313006 446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:22.313119 446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:22.320501 446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:22.320611 446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:22.320736 446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:06:22.329842 446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:22.331128 446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:22.332837 446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:22.345474 452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:22.345477 454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:22.345505 451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:22.346194 446 server_base.cc:1034] running on GCE node
I20260924 10:06:22.346771 446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:22.347769 446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:22.348975 446 hybrid_clock.cc:648] HybridClock initialized: now 1790244382348924 us; error 44 us; skew 500 ppm
I20260924 10:06:22.349282 446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:22.351519 446 webserver.cc:460] Webserver started at http://127.13.162.2:42609/ using document root <none> and password file <none>
I20260924 10:06:22.352206 446 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:22.352304 446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:22.352591 446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:22.355440 446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "78db24a8d956495fae92d18d34e3c1ea"
format_stamp: "Formatted at 2026-09-24 10:06:22 on dist-test-slave-jvsj"
server_key: "2ba12cbaf8bd74753d7407df448e4753"
server_key_iv: "6187874d7ba0bf602538375131a24173"
server_key_version: "encryptionkey@0"
I20260924 10:06:22.356186 446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "78db24a8d956495fae92d18d34e3c1ea"
format_stamp: "Formatted at 2026-09-24 10:06:22 on dist-test-slave-jvsj"
server_key: "2ba12cbaf8bd74753d7407df448e4753"
server_key_iv: "6187874d7ba0bf602538375131a24173"
server_key_version: "encryptionkey@0"
I20260924 10:06:22.361128 446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:22.364346 460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:22.365556 446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:06:22.365692 446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "78db24a8d956495fae92d18d34e3c1ea"
format_stamp: "Formatted at 2026-09-24 10:06:22 on dist-test-slave-jvsj"
server_key: "2ba12cbaf8bd74753d7407df448e4753"
server_key_iv: "6187874d7ba0bf602538375131a24173"
server_key_version: "encryptionkey@0"
I20260924 10:06:22.365875 446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:22.389413 446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:22.390285 446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:22.390623 446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:22.391628 446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:22.393174 446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:22.393267 446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:22.393373 446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:22.393440 446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:22.423198 446 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:40335
I20260924 10:06:22.423255 572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:40335 every 8 connection(s)
I20260924 10:06:22.424511 446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:06:22.429261 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 446
I20260924 10:06:22.429441 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:06:22.429988 13960 external_mini_cluster.cc:1428] Setting key 018b0690d2975e5f175e2df56ea46d79
I20260924 10:06:22.433113 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33625
--builtin_ntp_servers=127.13.162.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:06:22.438613 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.427201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:51453 (local address 127.13.162.62:33625)
0924 10:06:22.427362 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:22.427366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:22.428787 (+ 1421us) server_negotiation.cc:407] Connection header received
0924 10:06:22.431144 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:22.431148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:22.431222 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:22.431304 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:22.432711 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.433482 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.435752 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.436211 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.436332 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.437903 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:22.437909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:22.437912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:22.437933 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:22.437965 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:22.437977 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:22.438041 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:22.438264 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:06:22.438377 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:06:22.440080 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.427672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33625 (local address 127.13.162.2:51453)
0924 10:06:22.428625 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:22.428669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:22.430881 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:22.431582 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:22.431605 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:22.431742 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:22.432504 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.432533 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.433648 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.433655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.435610 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.435627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.436443 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.436449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.436780 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.437522 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:22.437526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:22.437582 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:22.437707 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:22.438056 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:22.438067 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:22.438074 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:06:22.438260 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:06:22.438750 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":255,"threads_started":1}
I20260924 10:06:22.442515 573 heartbeater.cc:347] Connected to a master server at 127.13.162.62:33625
I20260924 10:06:22.442826 573 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:22.443565 573 heartbeater.cc:511] Master 127.13.162.62:33625 requested a full tablet report, sending...
I20260924 10:06:22.444970 32724 ts_manager.cc:194] Registered new tserver with Master: 78db24a8d956495fae92d18d34e3c1ea (127.13.162.2:40335)
I20260924 10:06:22.445828 32724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.2:51453
W20260924 10:06:22.762111 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:22.762414 576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:22.762543 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:22.769723 576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:22.769814 576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:22.769953 576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:06:22.778972 576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:33625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:22.780189 576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:22.781968 576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:22.794267 581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:22.794349 582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:22.794309 584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:22.795339 576 server_base.cc:1034] running on GCE node
I20260924 10:06:22.795962 576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:22.796941 576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:22.798171 576 hybrid_clock.cc:648] HybridClock initialized: now 1790244382798118 us; error 46 us; skew 500 ppm
I20260924 10:06:22.798517 576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:22.800678 576 webserver.cc:460] Webserver started at http://127.13.162.3:40583/ using document root <none> and password file <none>
I20260924 10:06:22.801342 576 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:22.801436 576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:22.801730 576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:22.804560 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "1c881278616b42458019280ef9801852"
format_stamp: "Formatted at 2026-09-24 10:06:22 on dist-test-slave-jvsj"
server_key: "dde711046cabafeedb35ff56911e3d65"
server_key_iv: "4a8c441dd27a68c046df5e3c84cfec2f"
server_key_version: "encryptionkey@0"
I20260924 10:06:22.805305 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "1c881278616b42458019280ef9801852"
format_stamp: "Formatted at 2026-09-24 10:06:22 on dist-test-slave-jvsj"
server_key: "dde711046cabafeedb35ff56911e3d65"
server_key_iv: "4a8c441dd27a68c046df5e3c84cfec2f"
server_key_version: "encryptionkey@0"
I20260924 10:06:22.810308 576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:06:22.813716 590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:22.814985 576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:06:22.815140 576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "1c881278616b42458019280ef9801852"
format_stamp: "Formatted at 2026-09-24 10:06:22 on dist-test-slave-jvsj"
server_key: "dde711046cabafeedb35ff56911e3d65"
server_key_iv: "4a8c441dd27a68c046df5e3c84cfec2f"
server_key_version: "encryptionkey@0"
I20260924 10:06:22.815330 576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:22.835393 576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:22.836217 576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:22.836513 576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:22.837684 576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:22.839288 576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:22.839380 576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:22.839488 576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:22.839564 576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:22.867148 576 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:38603
I20260924 10:06:22.867223 702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:38603 every 8 connection(s)
I20260924 10:06:22.868419 576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:06:22.873144 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 576
I20260924 10:06:22.873299 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:06:22.873706 13960 external_mini_cluster.cc:1428] Setting key f7cd3b2e468185c4f11fd57cbb34174f
I20260924 10:06:22.880851 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.870954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:40185 (local address 127.13.162.62:33625)
0924 10:06:22.871122 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:22.871127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:22.872488 (+ 1361us) server_negotiation.cc:407] Connection header received
0924 10:06:22.874390 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:22.874396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:22.874478 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:22.874581 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:22.875989 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.876900 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.878448 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.878865 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.878957 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.880160 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:22.880165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:22.880169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:22.880187 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:22.880217 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:22.880228 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:22.880295 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:22.880450 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 10:06:22.880557 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:06:22.881829 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.871435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:33625 (local address 127.13.162.3:40185)
0924 10:06:22.872337 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:22.872371 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:22.874027 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:22.874877 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:22.874889 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:22.875003 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:22.875777 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.875804 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.877046 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.877050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.878283 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.878295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.879045 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.879051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.879267 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.879877 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:22.879880 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:22.879919 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:22.880001 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:22.880298 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:22.880304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:22.880309 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:06:22.880437 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 10:06:22.880827 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":278,"threads_started":1}
I20260924 10:06:22.883134 703 heartbeater.cc:347] Connected to a master server at 127.13.162.62:33625
I20260924 10:06:22.883422 703 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:22.884187 703 heartbeater.cc:511] Master 127.13.162.62:33625 requested a full tablet report, sending...
I20260924 10:06:22.885567 32724 ts_manager.cc:194] Registered new tserver with Master: 1c881278616b42458019280ef9801852 (127.13.162.3:38603)
I20260924 10:06:22.886418 32724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.3:40185
I20260924 10:06:22.888831 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:06:22.898483 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.892868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.13.162.62:33625)
0924 10:06:22.893070 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:22.893075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:22.893441 (+ 366us) server_negotiation.cc:407] Connection header received
0924 10:06:22.893549 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:22.893552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:22.893622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:22.893702 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:22.894086 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.894838 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.896506 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.896884 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.896974 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.897871 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:22.897876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:22.897879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:22.897895 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:22.897925 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:22.897936 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:22.898038 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:22.898171 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 10:06:22.898213 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:06:22.902082 32724 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:06:22.908722 32724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:06:22.940284 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.920318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:40335 (local address 127.0.0.1:50642)
0924 10:06:22.931363 (+ 11045us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:22.931417 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:22.931621 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:22.932419 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:22.932427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:22.932462 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:22.933967 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.933979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.935674 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.935680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.937437 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.937450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.939294 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.939300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.939516 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.939567 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:06:22.939716 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:06:22.939904 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10753,"mutex_wait_us":24,"thread_start_us":10617,"threads_started":1}
I20260924 10:06:22.940304 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.920677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.13.162.2:40335)
0924 10:06:22.930776 (+ 10099us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:22.930785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:22.931824 (+ 1039us) server_negotiation.cc:407] Connection header received
0924 10:06:22.931931 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:22.931940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:22.932207 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:22.932389 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:22.934162 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.935525 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.937656 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.938754 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.938880 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.939150 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:22.939769 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 10:06:22.939943 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9912,"thread_start_us":9618,"threads_started":1}
I20260924 10:06:22.941924 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.920495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:35865 (local address 127.0.0.1:40244)
0924 10:06:22.933533 (+ 13038us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:22.933551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:22.933677 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:22.935143 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:22.935148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:22.935173 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:22.935502 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.935515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.937355 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.937360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.939310 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.939321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.941210 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.941216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.941379 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.941405 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:06:22.941478 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:06:22.941594 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12911,"mutex_wait_us":57,"thread_start_us":12739,"threads_started":1}
I20260924 10:06:22.942014 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.921091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.13.162.1:35865)
0924 10:06:22.934574 (+ 13483us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:22.934583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:22.934622 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:06:22.934719 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:22.934727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:22.934943 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:22.935695 (+ 752us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:22.935878 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.937208 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.939484 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.940779 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.940895 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.941079 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:22.941539 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 10:06:22.941704 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13307,"thread_start_us":13147,"threads_started":1}
I20260924 10:06:22.943089 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.921030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:38603 (local address 127.0.0.1:38352)
0924 10:06:22.934892 (+ 13862us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:22.934909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:22.935025 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:22.935567 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:22.935571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:22.935596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:22.936091 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.936103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.937708 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.937713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.940365 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:22.940380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.942365 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.942371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:22.942539 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.942570 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:06:22.942655 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:06:22.942770 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13748,"mutex_wait_us":47,"thread_start_us":13622,"threads_started":1}
I20260924 10:06:22.943224 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:22.921623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.13.162.3:38603)
0924 10:06:22.933081 (+ 11458us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:22.933091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:22.935021 (+ 1930us) server_negotiation.cc:407] Connection header received
0924 10:06:22.935157 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:22.935163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:22.935371 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:22.935560 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:22.936282 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.937600 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.940581 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:22.941831 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:22.941957 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:22.942229 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:22.942728 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 10:06:22.942883 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11294,"thread_start_us":11148,"threads_started":1}
I20260924 10:06:22.945026 508 tablet_service.cc:1467] Processing CreateTablet for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 (DEFAULT_TABLE table=test-table [id=62890267496547eea31ab7ab91fbef19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:22.945894 377 tablet_service.cc:1467] Processing CreateTablet for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 (DEFAULT_TABLE table=test-table [id=62890267496547eea31ab7ab91fbef19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:22.946468 508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:22.947266 377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:22.954452 638 tablet_service.cc:1467] Processing CreateTablet for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 (DEFAULT_TABLE table=test-table [id=62890267496547eea31ab7ab91fbef19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:22.955988 638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:22.959158 721 tablet_bootstrap.cc:492] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Bootstrap starting.
I20260924 10:06:22.961668 722 tablet_bootstrap.cc:492] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Bootstrap starting.
I20260924 10:06:22.962826 721 tablet_bootstrap.cc:654] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:22.964062 721 log.cc:824] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:22.965054 722 tablet_bootstrap.cc:654] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:22.966200 722 log.cc:824] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:22.967159 721 tablet_bootstrap.cc:492] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: No bootstrap required, opened a new log
I20260924 10:06:22.967406 721 ts_tablet_manager.cc:1397] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:06:22.969175 722 tablet_bootstrap.cc:492] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: No bootstrap required, opened a new log
I20260924 10:06:22.969473 722 ts_tablet_manager.cc:1397] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:06:22.969488 725 tablet_bootstrap.cc:492] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Bootstrap starting.
I20260924 10:06:22.972755 725 tablet_bootstrap.cc:654] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:22.973670 725 log.cc:824] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:22.976207 725 tablet_bootstrap.cc:492] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: No bootstrap required, opened a new log
I20260924 10:06:22.976441 725 ts_tablet_manager.cc:1397] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:06:22.978091 721 raft_consensus.cc:348] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:22.978574 721 raft_consensus.cc:374] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:22.978678 721 raft_consensus.cc:729] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78db24a8d956495fae92d18d34e3c1ea, State: Initialized, Role: FOLLOWER
I20260924 10:06:22.979300 721 consensus_queue.cc:260] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:22.979895 722 raft_consensus.cc:348] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:22.980433 722 raft_consensus.cc:374] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:22.980705 722 raft_consensus.cc:729] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce49464c72b4db3a0c114152448aa8b, State: Initialized, Role: FOLLOWER
I20260924 10:06:22.981141 721 ts_tablet_manager.cc:1428] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 10:06:22.981400 722 consensus_queue.cc:260] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:22.981720 573 heartbeater.cc:503] Master 127.13.162.62:33625 was elected leader, sending a full tablet report...
I20260924 10:06:22.985847 725 raft_consensus.cc:348] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:22.986274 725 raft_consensus.cc:374] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:22.986471 725 raft_consensus.cc:729] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c881278616b42458019280ef9801852, State: Initialized, Role: FOLLOWER
I20260924 10:06:22.987202 725 consensus_queue.cc:260] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:22.989333 442 heartbeater.cc:503] Master 127.13.162.62:33625 was elected leader, sending a full tablet report...
I20260924 10:06:22.990206 722 ts_tablet_manager.cc:1428] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Time spent starting tablet: real 0.021s user 0.008s sys 0.011s
I20260924 10:06:22.992949 703 heartbeater.cc:503] Master 127.13.162.62:33625 was elected leader, sending a full tablet report...
I20260924 10:06:22.992942 725 ts_tablet_manager.cc:1428] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
W20260924 10:06:23.120025 704 tablet.cc:2369] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:23.176456 574 tablet.cc:2369] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:23.236496 443 tablet.cc:2369] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:23.304338 727 raft_consensus.cc:482] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:06:23.304608 727 raft_consensus.cc:504] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:23.306347 727 leader_election.cc:288] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce49464c72b4db3a0c114152448aa8b (127.13.162.1:35865), 1c881278616b42458019280ef9801852 (127.13.162.3:38603)
I20260924 10:06:23.313382 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.306908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:56405 (local address 127.13.162.1:35865)
0924 10:06:23.307106 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:23.307112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:23.307791 (+ 679us) server_negotiation.cc:407] Connection header received
0924 10:06:23.308023 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:23.308028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:23.308109 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:23.308200 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:23.308897 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.309663 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.311730 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.312524 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.312667 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.312736 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:23.313053 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 10:06:23.313187 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:06:23.313407 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.307064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:48729 (local address 127.13.162.3:38603)
0924 10:06:23.307236 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:23.307241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:23.307857 (+ 616us) server_negotiation.cc:407] Connection header received
0924 10:06:23.308069 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:23.308073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:23.308132 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:23.308214 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:23.308944 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.309677 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.311700 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.312490 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.312640 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.312712 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:23.312990 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:06:23.313132 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:06:23.313470 731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.306867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:38603 (local address 127.13.162.2:48729)
0924 10:06:23.307760 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:23.307775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:23.307864 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:23.308250 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:23.308254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:23.308310 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:23.308789 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:23.308800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.309807 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.309811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:23.311512 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:23.311528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.312679 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.312686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:23.312827 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.312850 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:06:23.312945 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:06:23.313076 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":183,"spinlock_wait_cycles":38912,"thread_start_us":473,"threads_started":1}
I20260924 10:06:23.314009 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.306867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:35865 (local address 127.13.162.2:56405)
0924 10:06:23.307668 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:23.307688 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:23.307854 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:23.308242 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:23.308247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:23.308306 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:23.308746 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:23.308760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.309793 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.309798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:23.311541 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:23.311554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.312719 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.312725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:23.312869 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.312902 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:06:23.313583 (+ 681us) client_negotiation.cc:240] Negotiation successful
0924 10:06:23.313721 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":7,"spinlock_wait_cycles":25216,"thread_start_us":421,"threads_started":1}
I20260924 10:06:23.314720 658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117d29b5ad6c40c8beacf6ac2e1b4cd1" candidate_uuid: "78db24a8d956495fae92d18d34e3c1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c881278616b42458019280ef9801852" is_pre_election: true
I20260924 10:06:23.314980 397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117d29b5ad6c40c8beacf6ac2e1b4cd1" candidate_uuid: "78db24a8d956495fae92d18d34e3c1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce49464c72b4db3a0c114152448aa8b" is_pre_election: true
I20260924 10:06:23.315213 658 raft_consensus.cc:2454] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78db24a8d956495fae92d18d34e3c1ea in term 0.
I20260924 10:06:23.315498 397 raft_consensus.cc:2454] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78db24a8d956495fae92d18d34e3c1ea in term 0.
I20260924 10:06:23.315869 461 leader_election.cc:302] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c881278616b42458019280ef9801852, 78db24a8d956495fae92d18d34e3c1ea; no voters:
I20260924 10:06:23.316293 727 raft_consensus.cc:2781] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:06:23.316421 727 raft_consensus.cc:482] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:06:23.316530 727 raft_consensus.cc:3037] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:23.318768 727 raft_consensus.cc:504] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:23.319399 727 leader_election.cc:288] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [CANDIDATE]: Term 1 election: Requested vote from peers 7ce49464c72b4db3a0c114152448aa8b (127.13.162.1:35865), 1c881278616b42458019280ef9801852 (127.13.162.3:38603)
I20260924 10:06:23.319887 397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117d29b5ad6c40c8beacf6ac2e1b4cd1" candidate_uuid: "78db24a8d956495fae92d18d34e3c1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce49464c72b4db3a0c114152448aa8b"
I20260924 10:06:23.319914 658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117d29b5ad6c40c8beacf6ac2e1b4cd1" candidate_uuid: "78db24a8d956495fae92d18d34e3c1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c881278616b42458019280ef9801852"
I20260924 10:06:23.320086 658 raft_consensus.cc:3037] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:23.320111 397 raft_consensus.cc:3037] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:23.322300 658 raft_consensus.cc:2454] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78db24a8d956495fae92d18d34e3c1ea in term 1.
I20260924 10:06:23.322306 397 raft_consensus.cc:2454] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78db24a8d956495fae92d18d34e3c1ea in term 1.
I20260924 10:06:23.322726 463 leader_election.cc:302] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78db24a8d956495fae92d18d34e3c1ea, 7ce49464c72b4db3a0c114152448aa8b; no voters:
I20260924 10:06:23.323047 727 raft_consensus.cc:2781] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:23.323453 727 raft_consensus.cc:686] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 1 LEADER]: Becoming Leader. State: Replica: 78db24a8d956495fae92d18d34e3c1ea, State: Running, Role: LEADER
I20260924 10:06:23.323945 727 consensus_queue.cc:237] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } }
I20260924 10:06:23.328701 32724 catalog_manager.cc:5495] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 78db24a8d956495fae92d18d34e3c1ea (127.13.162.2). New cstate: current_term: 1 leader_uuid: "78db24a8d956495fae92d18d34e3c1ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78db24a8d956495fae92d18d34e3c1ea" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 } health_report { overall_health: UNKNOWN } } }
I20260924 10:06:23.387852 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.382423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.13.162.2:40335)
0924 10:06:23.382604 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:23.382611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:23.382745 (+ 134us) server_negotiation.cc:407] Connection header received
0924 10:06:23.382960 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:23.382965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:23.383054 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:23.383188 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:06:23.383560 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.384291 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.385711 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.386110 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.386219 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.386814 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:06:23.387294 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:06:23.387447 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:23.387584 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 10:06:23.387641 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:06:23.396637 658 raft_consensus.cc:1261] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 1 FOLLOWER]: Refusing update from remote peer 78db24a8d956495fae92d18d34e3c1ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:23.396675 397 raft_consensus.cc:1261] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 1 FOLLOWER]: Refusing update from remote peer 78db24a8d956495fae92d18d34e3c1ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:23.397667 732 consensus_queue.cc:1035] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce49464c72b4db3a0c114152448aa8b" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:23.397984 727 consensus_queue.cc:1035] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c881278616b42458019280ef9801852" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 38603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:23.425711 32723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:23.425987 32723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:23.430106 32723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b016ffa56ab44725a91c48c1827b5f9f: Sending DeleteTablet for 3 replicas of tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1
I20260924 10:06:23.431357 508 tablet_service.cc:1514] Processing DeleteTablet for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:23 UTC) from {username='slave'} at 127.0.0.1:50642
I20260924 10:06:23.431387 377 tablet_service.cc:1514] Processing DeleteTablet for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:23 UTC) from {username='slave'} at 127.0.0.1:40244
I20260924 10:06:23.431782 638 tablet_service.cc:1514] Processing DeleteTablet for tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:23 UTC) from {username='slave'} at 127.0.0.1:38352
I20260924 10:06:23.433037 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 315
I20260924 10:06:23.451632 747 tablet_replica.cc:330] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: stopping tablet replica
I20260924 10:06:23.452095 747 raft_consensus.cc:2229] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:23.452636 747 raft_consensus.cc:2258] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:23.455142 746 tablet_replica.cc:330] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: stopping tablet replica
I20260924 10:06:23.455276 747 ts_tablet_manager.cc:1905] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:23.456012 745 tablet_replica.cc:330] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: stopping tablet replica
I20260924 10:06:23.456835 745 raft_consensus.cc:2229] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:23.457378 746 raft_consensus.cc:2229] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:06:23.458621 746 raft_consensus.cc:2258] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:23.459316 745 raft_consensus.cc:2258] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:23.462492 747 ts_tablet_manager.cc:1918] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:23.462709 747 log.cc:1196] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/117d29b5ad6c40c8beacf6ac2e1b4cd1
I20260924 10:06:23.463239 747 ts_tablet_manager.cc:1939] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 1c881278616b42458019280ef9801852: Deleting consensus metadata
I20260924 10:06:23.465243 32708 catalog_manager.cc:4841] TS 1c881278616b42458019280ef9801852 (127.13.162.3:38603): tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 (table test-table [id=62890267496547eea31ab7ab91fbef19]) successfully deleted
I20260924 10:06:23.467810 745 ts_tablet_manager.cc:1905] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:23.469733 746 ts_tablet_manager.cc:1905] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:23.475104 745 ts_tablet_manager.cc:1918] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:23.475356 745 log.cc:1196] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/117d29b5ad6c40c8beacf6ac2e1b4cd1
I20260924 10:06:23.475943 745 ts_tablet_manager.cc:1939] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 7ce49464c72b4db3a0c114152448aa8b: Deleting consensus metadata
I20260924 10:06:23.478006 32710 catalog_manager.cc:4841] TS 7ce49464c72b4db3a0c114152448aa8b (127.13.162.1:35865): tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 (table test-table [id=62890267496547eea31ab7ab91fbef19]) successfully deleted
I20260924 10:06:23.479220 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.435750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.13.162.1:35865)
0924 10:06:23.435939 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:23.435946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:23.461562 (+ 25616us) server_negotiation.cc:407] Connection header received
0924 10:06:23.462959 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:23.462966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:23.463074 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:23.463148 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:23.463813 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.464870 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.466915 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.467564 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.467654 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.477485 (+ 9831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:23.477497 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:23.477516 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:23.478284 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:23.478383 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:23.478410 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:23.478521 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:23.478724 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 10:06:23.478871 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:06:23.479194 746 ts_tablet_manager.cc:1918] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:23.479378 746 log.cc:1196] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/117d29b5ad6c40c8beacf6ac2e1b4cd1
I20260924 10:06:23.480007 746 ts_tablet_manager.cc:1939] T 117d29b5ad6c40c8beacf6ac2e1b4cd1 P 78db24a8d956495fae92d18d34e3c1ea: Deleting consensus metadata
I20260924 10:06:23.480165 437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:23.481784 32711 catalog_manager.cc:4841] TS 78db24a8d956495fae92d18d34e3c1ea (127.13.162.2:40335): tablet 117d29b5ad6c40c8beacf6ac2e1b4cd1 (table test-table [id=62890267496547eea31ab7ab91fbef19]) successfully deleted
I20260924 10:06:23.569675 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 315
I20260924 10:06:23.584409 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 446
I20260924 10:06:23.592458 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.585190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.13.162.2:40335)
0924 10:06:23.585382 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:23.585389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:23.585587 (+ 198us) server_negotiation.cc:407] Connection header received
0924 10:06:23.585772 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:23.585776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:23.585879 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:23.585962 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:23.586493 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.587413 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.589236 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.589645 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.589738 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.590973 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:23.590982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:23.590997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:23.591685 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:23.591745 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:23.591777 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:23.591863 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:23.592007 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 10:06:23.592093 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:06:23.593250 568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:23.704097 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 446
I20260924 10:06:23.719080 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 576
I20260924 10:06:23.725833 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:23.719898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.13.162.3:38603)
0924 10:06:23.720127 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:23.720133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:23.720156 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:06:23.720390 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:23.720395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:23.720489 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:23.720581 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:23.720982 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.721693 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.723013 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:23.723431 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:23.723538 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:23.724516 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:23.724525 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:23.724539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:23.725096 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:23.725142 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:23.725162 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:23.725256 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:23.725388 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 10:06:23.725499 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:06:23.726625 698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:23.813380 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 576
I20260924 10:06:23.826297 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 32693
I20260924 10:06:23.827365 32754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:23.933516 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 32693
2026-09-24T10:06:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2989 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:24 dist-test-slave-jvsj krb5kdc[755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:24 dist-test-slave-jvsj krb5kdc[755](info): set up 2 sockets
Sep 24 10:06:24 dist-test-slave-jvsj krb5kdc[755](info): commencing operation
krb5kdc: starting...
W20260924 10:06:24.962370 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:25 dist-test-slave-jvsj krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:25.201967 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:40339
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:40339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:25.543099 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:25.543403 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:25.543533 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:25.551360 768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:25.551501 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:25.551687 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:25.551854 768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:25.551904 768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:25.562351 768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:40339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:25.563640 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:25.564968 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:25.576108 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:25.576210 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:25.576182 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:25.576979 768 server_base.cc:1034] running on GCE node
I20260924 10:06:25.577741 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:25.579195 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:25.580391 768 hybrid_clock.cc:648] HybridClock initialized: now 1790244385580339 us; error 42 us; skew 500 ppm
I20260924 10:06:25.580778 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:25 dist-test-slave-jvsj krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244385, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:25.588348 768 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:06:25.590071 768 webserver.cc:460] Webserver started at http://127.13.162.62:35263/ using document root <none> and password file <none>
I20260924 10:06:25.590874 768 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:25.590996 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:25.591272 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:25.594609 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "a3856e405b12434597de0f226bf35fed"
format_stamp: "Formatted at 2026-09-24 10:06:25 on dist-test-slave-jvsj"
server_key: "405a1d922bd390986a81fcc12affc71e"
server_key_iv: "ad538df09bf6a67311ec9e4b195a9479"
server_key_version: "encryptionkey@0"
I20260924 10:06:25.595376 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "a3856e405b12434597de0f226bf35fed"
format_stamp: "Formatted at 2026-09-24 10:06:25 on dist-test-slave-jvsj"
server_key: "405a1d922bd390986a81fcc12affc71e"
server_key_iv: "ad538df09bf6a67311ec9e4b195a9479"
server_key_version: "encryptionkey@0"
I20260924 10:06:25.600836 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:25.604401 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:25.605773 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:06:25.605916 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "a3856e405b12434597de0f226bf35fed"
format_stamp: "Formatted at 2026-09-24 10:06:25 on dist-test-slave-jvsj"
server_key: "405a1d922bd390986a81fcc12affc71e"
server_key_iv: "ad538df09bf6a67311ec9e4b195a9479"
server_key_version: "encryptionkey@0"
I20260924 10:06:25.606079 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:25.619145 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:25.623497 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:25.623752 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:25.641975 768 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:40339
I20260924 10:06:25.641971 834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:40339 every 8 connection(s)
I20260924 10:06:25.643419 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:25.645810 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 768
I20260924 10:06:25.645990 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:25.646571 13960 external_mini_cluster.cc:1428] Setting key 6a7037b801f9bab240abd6eb00d5ed34
I20260924 10:06:25.647722 835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:06:25 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:25.662627 835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed: Bootstrap starting.
I20260924 10:06:25.666056 835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:25.667410 835 log.cc:824] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:25.670499 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:25.648414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.13.162.62:40339)
0924 10:06:25.649185 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:25.649198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:25.649240 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:06:25.650886 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:25.650910 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:25.651350 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:25.651708 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:25.653466 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:25.655050 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:25.656876 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:25.657578 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:25.657728 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:25.662757 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:25.662769 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:25.662790 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:25.662845 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:25.665808 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:25.666823 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:25.666831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:25.666838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:25.667026 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:25.667474 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:25.667478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:25.667480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:25.668201 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:25.668519 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:25.668908 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 10:06:25.669368 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":335,"threads_started":1}
I20260924 10:06:25.670899 835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed: No bootstrap required, opened a new log
I20260924 10:06:25.678663 835 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3856e405b12434597de0f226bf35fed" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 40339 } }
I20260924 10:06:25.679090 835 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:25.679190 835 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3856e405b12434597de0f226bf35fed, State: Initialized, Role: FOLLOWER
I20260924 10:06:25.679785 835 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3856e405b12434597de0f226bf35fed" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 40339 } }
I20260924 10:06:25.680012 835 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:25.680135 835 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:25.680296 835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:25.682474 835 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3856e405b12434597de0f226bf35fed" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 40339 } }
I20260924 10:06:25.682983 835 leader_election.cc:302] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3856e405b12434597de0f226bf35fed; no voters:
I20260924 10:06:25.683535 835 leader_election.cc:288] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:25.683676 840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:25.684300 840 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [term 1 LEADER]: Becoming Leader. State: Replica: a3856e405b12434597de0f226bf35fed, State: Running, Role: LEADER
I20260924 10:06:25.685204 840 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3856e405b12434597de0f226bf35fed" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 40339 } }
I20260924 10:06:25.685667 835 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:25.688172 842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3856e405b12434597de0f226bf35fed. Latest consensus state: current_term: 1 leader_uuid: "a3856e405b12434597de0f226bf35fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3856e405b12434597de0f226bf35fed" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 40339 } } }
I20260924 10:06:25.688581 842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:25.688515 841 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3856e405b12434597de0f226bf35fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3856e405b12434597de0f226bf35fed" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 40339 } } }
I20260924 10:06:25.688795 841 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:25.690231 849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:25.695093 849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:25.703575 849 catalog_manager.cc:1347] Generated new cluster ID: babcd405fdc94dccbd60693e3c78a423
I20260924 10:06:25.703693 849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:25.716969 849 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:25.718353 849 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:25.724798 849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed: Generated new TSK 0
I20260924 10:06:25.725914 849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.162.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
Entry for principal kudu/127.13.162.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab.
I20260924 10:06:25.791870 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:40339
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:26.121316 861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:26.121621 861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:26.121768 861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:26.129081 861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:26.129215 861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:26.129369 861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:06:26.139269 861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.1.keytab
--principal=kudu/127.13.162.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:40339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:26.140602 861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:26.141953 861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:26.155287 866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:26.155489 869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:26.155342 867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:26.156388 861 server_base.cc:1034] running on GCE node
I20260924 10:06:26.157056 861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:26.158138 861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:26.159353 861 hybrid_clock.cc:648] HybridClock initialized: now 1790244386159298 us; error 68 us; skew 500 ppm
I20260924 10:06:26.159677 861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:26 dist-test-slave-jvsj krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:26.162897 861 init.cc:377] Logged in from keytab as kudu/127.13.162.1@KRBTEST.COM (short username kudu)
I20260924 10:06:26.164386 861 webserver.cc:460] Webserver started at http://127.13.162.1:38961/ using document root <none> and password file <none>
I20260924 10:06:26.165059 861 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:26.165167 861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:26.165477 861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:26.168457 861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "ec31d9da141947a6ad1f1544396042c1"
format_stamp: "Formatted at 2026-09-24 10:06:26 on dist-test-slave-jvsj"
server_key: "4ce54e5bc6d2c34ea6e2bc1b60de7d06"
server_key_iv: "bdb1c9e12857982fa97effad92fcb462"
server_key_version: "encryptionkey@0"
I20260924 10:06:26.169333 861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "ec31d9da141947a6ad1f1544396042c1"
format_stamp: "Formatted at 2026-09-24 10:06:26 on dist-test-slave-jvsj"
server_key: "4ce54e5bc6d2c34ea6e2bc1b60de7d06"
server_key_iv: "bdb1c9e12857982fa97effad92fcb462"
server_key_version: "encryptionkey@0"
I20260924 10:06:26.174615 861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:06:26.178192 876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:26.179591 861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:26.179752 861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "ec31d9da141947a6ad1f1544396042c1"
format_stamp: "Formatted at 2026-09-24 10:06:26 on dist-test-slave-jvsj"
server_key: "4ce54e5bc6d2c34ea6e2bc1b60de7d06"
server_key_iv: "bdb1c9e12857982fa97effad92fcb462"
server_key_version: "encryptionkey@0"
I20260924 10:06:26.179937 861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:26.199723 861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:26.204002 861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:26.204293 861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:26.205262 861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:26.206604 861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:26.206686 861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:26.206801 861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:26.206853 861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:26.236706 861 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:44429
I20260924 10:06:26.236753 988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:44429 every 8 connection(s)
I20260924 10:06:26.237982 861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:06:26.238408 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 861
I20260924 10:06:26.238626 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:06:26.239205 13960 external_mini_cluster.cc:1428] Setting key 66cf6471ecf8e9648cc896314af4572c
Sep 24 10:06:26 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.1@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:26.260327 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:26.240841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:54021 (local address 127.13.162.62:40339)
0924 10:06:26.241350 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:26.241356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:26.242497 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 10:06:26.244946 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:26.244951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:26.245018 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:26.245108 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:26.247044 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.247803 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.249868 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.250269 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.250415 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:26.253977 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:26.253982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:26.253986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:26.254022 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:26.256612 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:26.257524 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:26.257528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:26.257530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:26.257655 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:26.258165 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:26.258170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:26.258173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:26.258551 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:26.258743 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:26.259634 (+ 891us) server_negotiation.cc:299] Negotiation successful
0924 10:06:26.259954 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":274,"threads_started":1}
I20260924 10:06:26.261488 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:26.241474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:40339 (local address 127.13.162.1:54021)
0924 10:06:26.242302 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:26.242390 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:26.244650 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:26.245374 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:26.245385 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:26.246064 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:26.246858 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:26.246881 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.247945 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.247951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:26.249703 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:26.249718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.250513 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.250520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:26.250727 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:26.251345 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:26.251371 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:26.253726 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:26.256799 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:26.256808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:26.256827 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:26.257396 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:26.257790 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:26.257794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:26.257797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:26.258030 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:26.258687 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:26.258696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:26.259069 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 10:06:26.259428 (+ 359us) client_negotiation.cc:240] Negotiation successful
0924 10:06:26.260013 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":313,"threads_started":1}
I20260924 10:06:26.263279 989 heartbeater.cc:347] Connected to a master server at 127.13.162.62:40339
I20260924 10:06:26.263659 989 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:26.264592 989 heartbeater.cc:511] Master 127.13.162.62:40339 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.162.2@KRBTEST.COM; defaulting to no policy
I20260924 10:06:26.267132 800 ts_manager.cc:194] Registered new tserver with Master: ec31d9da141947a6ad1f1544396042c1 (127.13.162.1:44429)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.2@KRBTEST.COM" created.
I20260924 10:06:26.268669 800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.1@KRBTEST.COM'} at 127.13.162.1:54021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
Entry for principal kudu/127.13.162.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab.
I20260924 10:06:26.297215 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:40339
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:26.622743 995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:26.623034 995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:26.623136 995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:26.630309 995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:26.630465 995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:26.630586 995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:06:26.640470 995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.2.keytab
--principal=kudu/127.13.162.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:40339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:26.642002 995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:26.643726 995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:26.658569 1001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:26.658522 1003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:26.659013 1000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:26.659510 995 server_base.cc:1034] running on GCE node
I20260924 10:06:26.660128 995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:26.661124 995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:26.662423 995 hybrid_clock.cc:648] HybridClock initialized: now 1790244386662363 us; error 42 us; skew 500 ppm
I20260924 10:06:26.662742 995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:26 dist-test-slave-jvsj krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:26.665719 995 init.cc:377] Logged in from keytab as kudu/127.13.162.2@KRBTEST.COM (short username kudu)
I20260924 10:06:26.667261 995 webserver.cc:460] Webserver started at http://127.13.162.2:34329/ using document root <none> and password file <none>
I20260924 10:06:26.667935 995 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:26.668051 995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:26.668352 995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:26.671366 995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "c41d10ba629744ef8d7f31972fa4732e"
format_stamp: "Formatted at 2026-09-24 10:06:26 on dist-test-slave-jvsj"
server_key: "30d2f333572b710257829bb2a0190c22"
server_key_iv: "4074116a6b83de4d45d8cebe46416695"
server_key_version: "encryptionkey@0"
I20260924 10:06:26.672189 995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "c41d10ba629744ef8d7f31972fa4732e"
format_stamp: "Formatted at 2026-09-24 10:06:26 on dist-test-slave-jvsj"
server_key: "30d2f333572b710257829bb2a0190c22"
server_key_iv: "4074116a6b83de4d45d8cebe46416695"
server_key_version: "encryptionkey@0"
I20260924 10:06:26.677636 995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:06:26.681147 1010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:26.682515 995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:06:26.682662 995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "c41d10ba629744ef8d7f31972fa4732e"
format_stamp: "Formatted at 2026-09-24 10:06:26 on dist-test-slave-jvsj"
server_key: "30d2f333572b710257829bb2a0190c22"
server_key_iv: "4074116a6b83de4d45d8cebe46416695"
server_key_version: "encryptionkey@0"
I20260924 10:06:26.682854 995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:26.704401 995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:26.708823 995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:26.709097 995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:26.710191 995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:26.711673 995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:26.711773 995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:26.711900 995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:26.711952 995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:26.743888 995 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:40965
I20260924 10:06:26.743939 1122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:40965 every 8 connection(s)
I20260924 10:06:26.745260 995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:06:26.751078 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 995
I20260924 10:06:26.751289 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:06:26.751875 13960 external_mini_cluster.cc:1428] Setting key 1af8d9197d015b287da8b1988a332608
Sep 24 10:06:26 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.2@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:26.767877 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:26.747928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:52217 (local address 127.13.162.62:40339)
0924 10:06:26.748139 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:26.748144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:26.749358 (+ 1214us) server_negotiation.cc:407] Connection header received
0924 10:06:26.751548 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:26.751553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:26.751635 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:26.751732 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:26.753723 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.754597 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.756919 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.757366 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.757460 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:26.761622 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:26.761627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:26.761630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:26.761665 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:26.764610 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:26.765390 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:26.765396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:26.765400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:26.765554 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:26.766082 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:26.766087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:26.766090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:26.766474 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:26.766614 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:26.767332 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 10:06:26.767558 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 10:06:26.768828 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:26.748429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:40339 (local address 127.13.162.2:52217)
0924 10:06:26.749178 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:26.749220 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:26.751174 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:26.752039 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:26.752051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:26.752825 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:26.753536 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:26.753561 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.754790 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.754795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:26.756764 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:26.756781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:26.757593 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:26.757600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:26.757897 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:26.758677 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:26.758706 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:26.761362 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:26.764773 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:26.764781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:26.764795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:26.765275 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:26.765686 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:26.765690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:26.765693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:26.765977 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:26.766605 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:26.766612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:26.766973 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 10:06:26.767232 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:06:26.767642 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":293,"threads_started":1}
I20260924 10:06:26.770408 1123 heartbeater.cc:347] Connected to a master server at 127.13.162.62:40339
I20260924 10:06:26.770740 1123 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:26.771575 1123 heartbeater.cc:511] Master 127.13.162.62:40339 requested a full tablet report, sending...
I20260924 10:06:26.773288 800 ts_manager.cc:194] Registered new tserver with Master: c41d10ba629744ef8d7f31972fa4732e (127.13.162.2:40965)
I20260924 10:06:26.774410 800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.2@KRBTEST.COM'} at 127.13.162.2:52217
WARNING: no policy specified for kudu/127.13.162.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
Entry for principal kudu/127.13.162.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab.
I20260924 10:06:26.807371 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:40339
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:27.124593 1128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:27.124888 1128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:27.124984 1128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:27.132208 1128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:27.132321 1128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:27.132438 1128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:06:27.142387 1128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.3.keytab
--principal=kudu/127.13.162.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:40339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:27.143622 1128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:27.144876 1128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:27.157524 1133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:27.157558 1134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:27.157496 1136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:27.158442 1128 server_base.cc:1034] running on GCE node
I20260924 10:06:27.159029 1128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:27.159932 1128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:27.161124 1128 hybrid_clock.cc:648] HybridClock initialized: now 1790244387161081 us; error 39 us; skew 500 ppm
I20260924 10:06:27.161439 1128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:27 dist-test-slave-jvsj krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244387, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:27.164460 1128 init.cc:377] Logged in from keytab as kudu/127.13.162.3@KRBTEST.COM (short username kudu)
I20260924 10:06:27.165946 1128 webserver.cc:460] Webserver started at http://127.13.162.3:35973/ using document root <none> and password file <none>
I20260924 10:06:27.166656 1128 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:27.166766 1128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:27.167035 1128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:27.170035 1128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "c745e7792507454fbd68d0f973df8c8f"
format_stamp: "Formatted at 2026-09-24 10:06:27 on dist-test-slave-jvsj"
server_key: "2a655885f192aabf2c16b30e907fca99"
server_key_iv: "cbd1082f11e0ed4d43aaf635cbcca022"
server_key_version: "encryptionkey@0"
I20260924 10:06:27.170847 1128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "c745e7792507454fbd68d0f973df8c8f"
format_stamp: "Formatted at 2026-09-24 10:06:27 on dist-test-slave-jvsj"
server_key: "2a655885f192aabf2c16b30e907fca99"
server_key_iv: "cbd1082f11e0ed4d43aaf635cbcca022"
server_key_version: "encryptionkey@0"
I20260924 10:06:27.176110 1128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:27.179785 1143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:27.181208 1128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:27.181357 1128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "c745e7792507454fbd68d0f973df8c8f"
format_stamp: "Formatted at 2026-09-24 10:06:27 on dist-test-slave-jvsj"
server_key: "2a655885f192aabf2c16b30e907fca99"
server_key_iv: "cbd1082f11e0ed4d43aaf635cbcca022"
server_key_version: "encryptionkey@0"
I20260924 10:06:27.181546 1128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:27.197568 1128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:27.201587 1128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:27.201869 1128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:27.202944 1128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:27.204294 1128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:27.204365 1128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:27.204468 1128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:27.204516 1128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:27.236166 1128 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:34873
I20260924 10:06:27.236186 1255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:34873 every 8 connection(s)
I20260924 10:06:27.237949 1128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:06:27.239753 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1128
I20260924 10:06:27.239923 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:06:27.240464 13960 external_mini_cluster.cc:1428] Setting key 004f72afdbb88095063c9924ba55e0b3
Sep 24 10:06:27 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244387, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.3@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:27.259641 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.241416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:45621 (local address 127.13.162.62:40339)
0924 10:06:27.241605 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.241611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.243182 (+ 1571us) server_negotiation.cc:407] Connection header received
0924 10:06:27.245048 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.245053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.245120 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.245205 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:27.246900 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.247653 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.249311 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.249729 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.249840 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.253484 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:27.253490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:27.253493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:27.253529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:27.256619 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.257366 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.257371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.257375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.257505 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.257927 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.257932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.257934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.258201 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:27.258360 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.259072 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.259294 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:06:27.260632 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.242105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:40339 (local address 127.13.162.3:45621)
0924 10:06:27.242986 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.243045 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:27.244767 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:27.245512 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:27.245521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:27.246064 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:27.246718 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.246739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.247852 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.247858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.249156 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.249169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.249954 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.249961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.250205 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.250909 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:06:27.250932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:27.253217 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:27.256754 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:27.256765 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:27.256784 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:27.257243 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:27.257605 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:06:27.257610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:06:27.257614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:06:27.257815 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:06:27.258345 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:27.258354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:27.258710 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 10:06:27.258945 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 10:06:27.259506 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":350,"threads_started":1}
I20260924 10:06:27.262105 1256 heartbeater.cc:347] Connected to a master server at 127.13.162.62:40339
I20260924 10:06:27.262498 1256 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:27.263332 1256 heartbeater.cc:511] Master 127.13.162.62:40339 requested a full tablet report, sending...
I20260924 10:06:27.265017 800 ts_manager.cc:194] Registered new tserver with Master: c745e7792507454fbd68d0f973df8c8f (127.13.162.3:34873)
I20260924 10:06:27.265823 800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.162.3@KRBTEST.COM'} at 127.13.162.3:45621
I20260924 10:06:27.268368 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:06:27.272634 989 heartbeater.cc:503] Master 127.13.162.62:40339 was elected leader, sending a full tablet report...
I20260924 10:06:27.286038 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.272814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.13.162.62:40339)
0924 10:06:27.273029 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.273037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.273465 (+ 428us) server_negotiation.cc:407] Connection header received
0924 10:06:27.273660 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.273666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.273753 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.273864 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:27.275480 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.276219 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.277620 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.278034 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.278127 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.280673 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:27.280677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:27.280681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:27.280720 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:27.283269 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.284072 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.284077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.284079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.284240 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.284579 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.284581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.284584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.284951 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:27.285162 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.285501 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.285752 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:06:27.289997 800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:06:27.296928 800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:06:27.320415 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.309968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.13.162.2:40965)
0924 10:06:27.310682 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.310690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.311177 (+ 487us) server_negotiation.cc:407] Connection header received
0924 10:06:27.311473 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.311482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.311769 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.311951 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:27.313472 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.315181 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.317805 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.319214 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.319342 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.319530 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.319909 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.320090 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":444,"threads_started":1}
I20260924 10:06:27.320497 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.309693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:40965 (local address 127.0.0.1:35260)
0924 10:06:27.311079 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.311094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:27.311263 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:27.312250 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:27.312258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:27.312356 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:27.312810 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.312825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.315359 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.315367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.317600 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.317616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.319411 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.319417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.319620 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.319691 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:06:27.319829 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 10:06:27.320028 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"mutex_wait_us":31,"spinlock_wait_cycles":6656,"thread_start_us":1247,"threads_started":1}
I20260924 10:06:27.320577 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.309704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.13.162.1:44429)
0924 10:06:27.310353 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.310364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.311425 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 10:06:27.311519 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.311525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.311769 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.311951 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:27.313564 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.315144 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.317824 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.319179 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.319312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.319549 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.320097 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.320277 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":346,"threads_started":1}
I20260924 10:06:27.321504 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.312055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.13.162.3:34873)
0924 10:06:27.313039 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.313048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.313703 (+ 655us) server_negotiation.cc:407] Connection header received
0924 10:06:27.313827 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.313836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.314113 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.314344 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:27.314888 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.316316 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.318878 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.320159 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.320280 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.320485 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.321014 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.321186 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":645,"threads_started":1}
I20260924 10:06:27.321564 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.309439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59664)
0924 10:06:27.310947 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.311001 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:27.311217 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:27.312250 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:27.312258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:27.312308 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:27.313303 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.313320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.315310 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.315316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.317626 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.317640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.319739 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.319745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.319899 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.319928 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:06:27.320850 (+ 922us) client_negotiation.cc:240] Negotiation successful
0924 10:06:27.320982 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":545,"spinlock_wait_cycles":1280,"thread_start_us":1379,"threads_started":1}
I20260924 10:06:27.322585 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.311694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:34873 (local address 127.0.0.1:34230)
0924 10:06:27.313446 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.313466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:27.313582 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:27.314339 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:27.314345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:27.314371 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:27.314734 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.314743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.316477 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.316482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.318700 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.318716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.320644 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.320650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.320819 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.320850 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:06:27.320944 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:06:27.321075 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":23,"thread_start_us":1483,"threads_started":1}
I20260924 10:06:27.324919 1058 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb38c8b172d430e812d557bab1dae3d (DEFAULT_TABLE table=test-table [id=988b0ca34e3e4b9fb7aa69f75535c7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:27.325107 924 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb38c8b172d430e812d557bab1dae3d (DEFAULT_TABLE table=test-table [id=988b0ca34e3e4b9fb7aa69f75535c7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:27.326295 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb38c8b172d430e812d557bab1dae3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:27.326624 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb38c8b172d430e812d557bab1dae3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:27.326162 1191 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb38c8b172d430e812d557bab1dae3d (DEFAULT_TABLE table=test-table [id=988b0ca34e3e4b9fb7aa69f75535c7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:27.327703 1191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb38c8b172d430e812d557bab1dae3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:27.338021 1274 tablet_bootstrap.cc:492] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Bootstrap starting.
I20260924 10:06:27.341818 1274 tablet_bootstrap.cc:654] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:27.342686 1276 tablet_bootstrap.cc:492] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Bootstrap starting.
I20260924 10:06:27.343793 1275 tablet_bootstrap.cc:492] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Bootstrap starting.
I20260924 10:06:27.344079 1274 log.cc:824] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:27.346339 1276 tablet_bootstrap.cc:654] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:27.347210 1275 tablet_bootstrap.cc:654] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:27.347682 1274 tablet_bootstrap.cc:492] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: No bootstrap required, opened a new log
I20260924 10:06:27.347651 1276 log.cc:824] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:27.347990 1274 ts_tablet_manager.cc:1397] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 10:06:27.348251 1275 log.cc:824] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:27.351425 1276 tablet_bootstrap.cc:492] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: No bootstrap required, opened a new log
I20260924 10:06:27.351424 1275 tablet_bootstrap.cc:492] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: No bootstrap required, opened a new log
I20260924 10:06:27.351769 1275 ts_tablet_manager.cc:1397] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260924 10:06:27.351897 1276 ts_tablet_manager.cc:1397] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260924 10:06:27.356412 1274 raft_consensus.cc:348] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.356895 1274 raft_consensus.cc:374] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:27.357031 1274 raft_consensus.cc:729] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41d10ba629744ef8d7f31972fa4732e, State: Initialized, Role: FOLLOWER
I20260924 10:06:27.357683 1274 consensus_queue.cc:260] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.358850 1123 heartbeater.cc:503] Master 127.13.162.62:40339 was elected leader, sending a full tablet report...
I20260924 10:06:27.360929 1274 ts_tablet_manager.cc:1428] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 10:06:27.362900 1276 raft_consensus.cc:348] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.362900 1275 raft_consensus.cc:348] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.363394 1275 raft_consensus.cc:374] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:27.363435 1276 raft_consensus.cc:374] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:27.363528 1275 raft_consensus.cc:729] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec31d9da141947a6ad1f1544396042c1, State: Initialized, Role: FOLLOWER
I20260924 10:06:27.363536 1276 raft_consensus.cc:729] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c745e7792507454fbd68d0f973df8c8f, State: Initialized, Role: FOLLOWER
I20260924 10:06:27.364244 1276 consensus_queue.cc:260] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.364245 1275 consensus_queue.cc:260] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.365561 1256 heartbeater.cc:503] Master 127.13.162.62:40339 was elected leader, sending a full tablet report...
I20260924 10:06:27.366379 1276 ts_tablet_manager.cc:1428] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260924 10:06:27.366786 1275 ts_tablet_manager.cc:1428] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
W20260924 10:06:27.489961 1257 tablet.cc:2369] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:27.490653 990 tablet.cc:2369] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:27.497259 1124 tablet.cc:2369] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:27.506568 1280 raft_consensus.cc:482] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:06:27.506887 1280 raft_consensus.cc:504] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.508663 1280 leader_election.cc:288] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec31d9da141947a6ad1f1544396042c1 (127.13.162.1:44429), c745e7792507454fbd68d0f973df8c8f (127.13.162.3:34873)
I20260924 10:06:27.518092 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.509155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:37355 (local address 127.13.162.3:34873)
0924 10:06:27.509388 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.509396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.509974 (+ 578us) server_negotiation.cc:407] Connection header received
0924 10:06:27.510257 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.510262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.510366 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.510468 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:27.511273 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.512468 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.515553 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.516894 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.517041 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.517105 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.517677 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.517852 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:06:27.518096 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.509184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:33703 (local address 127.13.162.1:44429)
0924 10:06:27.509395 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.509401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.509904 (+ 503us) server_negotiation.cc:407] Connection header received
0924 10:06:27.510251 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.510257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.510366 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.510467 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:27.511273 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.512424 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.515550 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.516827 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.517019 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.517100 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.517677 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.517822 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:06:27.518385 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.509012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.13.162.2:33703)
0924 10:06:27.509756 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.509782 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:27.510063 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:27.510492 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:27.510498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:27.510528 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:27.511059 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.511078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.512612 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.512618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.515311 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.515334 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.517127 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.517134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.517461 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.517532 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:06:27.517624 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:06:27.517864 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":257,"thread_start_us":392,"threads_started":1}
I20260924 10:06:27.518409 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.509048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:34873 (local address 127.13.162.2:37355)
0924 10:06:27.509866 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.509882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:27.510117 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:27.510492 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:27.510498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:27.510542 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:27.511088 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.511097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.512612 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.512618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.515311 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:27.515334 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.517131 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.517136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:27.517464 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.517528 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 10:06:27.517624 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:06:27.517863 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":167,"spinlock_wait_cycles":56064,"thread_start_us":504,"threads_started":1}
I20260924 10:06:27.519739 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb38c8b172d430e812d557bab1dae3d" candidate_uuid: "c41d10ba629744ef8d7f31972fa4732e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c745e7792507454fbd68d0f973df8c8f" is_pre_election: true
I20260924 10:06:27.519739 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb38c8b172d430e812d557bab1dae3d" candidate_uuid: "c41d10ba629744ef8d7f31972fa4732e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec31d9da141947a6ad1f1544396042c1" is_pre_election: true
I20260924 10:06:27.520370 1211 raft_consensus.cc:2454] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c41d10ba629744ef8d7f31972fa4732e in term 0.
I20260924 10:06:27.520370 944 raft_consensus.cc:2454] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c41d10ba629744ef8d7f31972fa4732e in term 0.
I20260924 10:06:27.521169 1012 leader_election.cc:302] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c41d10ba629744ef8d7f31972fa4732e, ec31d9da141947a6ad1f1544396042c1; no voters:
I20260924 10:06:27.521622 1280 raft_consensus.cc:2781] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:06:27.521812 1280 raft_consensus.cc:482] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:06:27.521911 1280 raft_consensus.cc:3037] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:27.524128 1280 raft_consensus.cc:504] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.524760 1280 leader_election.cc:288] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [CANDIDATE]: Term 1 election: Requested vote from peers ec31d9da141947a6ad1f1544396042c1 (127.13.162.1:44429), c745e7792507454fbd68d0f973df8c8f (127.13.162.3:34873)
I20260924 10:06:27.525307 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb38c8b172d430e812d557bab1dae3d" candidate_uuid: "c41d10ba629744ef8d7f31972fa4732e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec31d9da141947a6ad1f1544396042c1"
I20260924 10:06:27.525312 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb38c8b172d430e812d557bab1dae3d" candidate_uuid: "c41d10ba629744ef8d7f31972fa4732e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c745e7792507454fbd68d0f973df8c8f"
I20260924 10:06:27.525501 1211 raft_consensus.cc:3037] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:27.525506 944 raft_consensus.cc:3037] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:27.527776 1211 raft_consensus.cc:2454] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c41d10ba629744ef8d7f31972fa4732e in term 1.
I20260924 10:06:27.527810 944 raft_consensus.cc:2454] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c41d10ba629744ef8d7f31972fa4732e in term 1.
I20260924 10:06:27.528260 1011 leader_election.cc:302] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c41d10ba629744ef8d7f31972fa4732e, c745e7792507454fbd68d0f973df8c8f; no voters:
I20260924 10:06:27.528616 1280 raft_consensus.cc:2781] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:27.529069 1280 raft_consensus.cc:686] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 1 LEADER]: Becoming Leader. State: Replica: c41d10ba629744ef8d7f31972fa4732e, State: Running, Role: LEADER
I20260924 10:06:27.529567 1280 consensus_queue.cc:237] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } }
I20260924 10:06:27.534694 800 catalog_manager.cc:5495] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e reported cstate change: term changed from 0 to 1, leader changed from <none> to c41d10ba629744ef8d7f31972fa4732e (127.13.162.2). New cstate: current_term: 1 leader_uuid: "c41d10ba629744ef8d7f31972fa4732e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41d10ba629744ef8d7f31972fa4732e" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 40965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 } health_report { overall_health: UNKNOWN } } }
I20260924 10:06:27.562776 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.557411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.13.162.2:40965)
0924 10:06:27.557617 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.557623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.557680 (+ 57us) server_negotiation.cc:407] Connection header received
0924 10:06:27.557913 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.557917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.558006 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.558139 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:06:27.558562 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.559250 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.560612 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.561041 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.561150 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.561640 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:06:27.562150 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:06:27.562287 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.562408 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.562496 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:06:27.571017 1211 raft_consensus.cc:1261] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 1 FOLLOWER]: Refusing update from remote peer c41d10ba629744ef8d7f31972fa4732e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:27.571017 944 raft_consensus.cc:1261] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 1 FOLLOWER]: Refusing update from remote peer c41d10ba629744ef8d7f31972fa4732e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:27.571969 1285 consensus_queue.cc:1035] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec31d9da141947a6ad1f1544396042c1" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:27.572314 1280 consensus_queue.cc:1035] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c745e7792507454fbd68d0f973df8c8f" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:27.585355 1289 mvcc.cc:205] Tried to move back new op lower bound from 7332841011480948736 to 7332841011325222912. Current Snapshot: MvccSnapshot[applied={T|T < 7332841011480948736}]
I20260924 10:06:27.596772 799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:27.597110 799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:27.602692 799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3856e405b12434597de0f226bf35fed: Sending DeleteTablet for 3 replicas of tablet 5cb38c8b172d430e812d557bab1dae3d
I20260924 10:06:27.603747 924 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb38c8b172d430e812d557bab1dae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:27 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:59664
I20260924 10:06:27.604224 1191 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb38c8b172d430e812d557bab1dae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:27 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:34230
I20260924 10:06:27.604450 1058 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb38c8b172d430e812d557bab1dae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:27 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:35260
I20260924 10:06:27.603762 923 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb38c8b172d430e812d557bab1dae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:27 UTC) from {username='kudu', principal='kudu/127.13.162.62@KRBTEST.COM'} at 127.0.0.1:59664
I20260924 10:06:27.605973 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 861
I20260924 10:06:27.618904 1300 tablet_replica.cc:330] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: stopping tablet replica
I20260924 10:06:27.619469 1300 raft_consensus.cc:2229] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:06:27.620177 1300 raft_consensus.cc:2258] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:27.621013 1299 tablet_replica.cc:330] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: stopping tablet replica
I20260924 10:06:27.621503 1299 raft_consensus.cc:2229] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:27.622035 1299 raft_consensus.cc:2258] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:27.624398 1299 ts_tablet_manager.cc:1905] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:27.624406 1300 ts_tablet_manager.cc:1905] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:27.631521 1299 ts_tablet_manager.cc:1918] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:27.631845 1299 log.cc:1196] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/5cb38c8b172d430e812d557bab1dae3d
I20260924 10:06:27.632524 1299 ts_tablet_manager.cc:1939] T 5cb38c8b172d430e812d557bab1dae3d P c745e7792507454fbd68d0f973df8c8f: Deleting consensus metadata
I20260924 10:06:27.632565 1298 tablet_replica.cc:330] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: stopping tablet replica
I20260924 10:06:27.633069 1298 raft_consensus.cc:2229] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:27.633630 1298 raft_consensus.cc:2258] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:27.634605 784 catalog_manager.cc:4841] TS c745e7792507454fbd68d0f973df8c8f (127.13.162.3:34873): tablet 5cb38c8b172d430e812d557bab1dae3d (table test-table [id=988b0ca34e3e4b9fb7aa69f75535c7f3]) successfully deleted
I20260924 10:06:27.636852 1298 ts_tablet_manager.cc:1905] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:27.640010 1300 ts_tablet_manager.cc:1918] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:27.640264 1300 log.cc:1196] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/5cb38c8b172d430e812d557bab1dae3d
I20260924 10:06:27.640874 1300 ts_tablet_manager.cc:1939] T 5cb38c8b172d430e812d557bab1dae3d P c41d10ba629744ef8d7f31972fa4732e: Deleting consensus metadata
Sep 24 10:06:27 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.1@KRBTEST.COM
I20260924 10:06:27.642912 784 catalog_manager.cc:4841] TS c41d10ba629744ef8d7f31972fa4732e (127.13.162.2:40965): tablet 5cb38c8b172d430e812d557bab1dae3d (table test-table [id=988b0ca34e3e4b9fb7aa69f75535c7f3]) successfully deleted
I20260924 10:06:27.649572 1298 ts_tablet_manager.cc:1918] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:27.649652 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.608955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.13.162.1:44429)
0924 10:06:27.609283 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.609291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.629679 (+ 20388us) server_negotiation.cc:407] Connection header received
0924 10:06:27.629992 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.629998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.630096 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.630178 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:27.634122 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.635108 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.636922 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.637581 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.637709 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.643372 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:27.643386 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:27.643400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:27.643444 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:27.646122 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.647006 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.647016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.647019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.647193 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.647651 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.647656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.647660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.648156 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:27.648452 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.649031 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.649276 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:06:27.649868 1298 log.cc:1196] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244085964582-13960-0/minicluster-data/ts-0/wal/wals/5cb38c8b172d430e812d557bab1dae3d
I20260924 10:06:27.650599 1298 ts_tablet_manager.cc:1939] T 5cb38c8b172d430e812d557bab1dae3d P ec31d9da141947a6ad1f1544396042c1: Deleting consensus metadata
I20260924 10:06:27.651326 984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:27.751242 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 861
I20260924 10:06:27.751353 785 catalog_manager.cc:4841] TS ec31d9da141947a6ad1f1544396042c1 (127.13.162.1:44429): tablet 5cb38c8b172d430e812d557bab1dae3d (table test-table [id=988b0ca34e3e4b9fb7aa69f75535c7f3]) successfully deleted
W20260924 10:06:27.773500 785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:44429 (error 108)
I20260924 10:06:27.774266 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 995
I20260924 10:06:27.775743 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.775213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59676)
0924 10:06:27.775458 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.775573 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260924 10:06:27.776014 785 catalog_manager.cc:4584] TS ec31d9da141947a6ad1f1544396042c1 (127.13.162.1:44429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cb38c8b172d430e812d557bab1dae3d: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Sep 24 10:06:27 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.2@KRBTEST.COM
I20260924 10:06:27.792165 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.775441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.13.162.2:40965)
0924 10:06:27.775638 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.775645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.775662 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:06:27.775722 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.775727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.775820 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.775938 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:27.777463 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.778856 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.780185 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.780852 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.780984 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.785238 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:27.785257 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:27.785279 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:27.785336 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:27.789076 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.790067 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.790076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.790078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.790227 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.790658 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.790661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.790666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.791103 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:27.791370 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.791650 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.791865 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:06:27.792961 1118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:27.829873 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.829290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59678)
0924 10:06:27.829548 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.829676 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 10:06:27.831003 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.830583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59682)
0924 10:06:27.830744 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.830849 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:06:27.891511 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.890951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59690)
0924 10:06:27.891174 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.891294 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:06:27.892624 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.892242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59704)
0924 10:06:27.892390 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.892482 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:06:27.910797 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 995
I20260924 10:06:27.926255 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1128
Sep 24 10:06:27 dist-test-slave-jvsj krb5kdc[755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.3@KRBTEST.COM
I20260924 10:06:27.940404 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.927009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.13.162.3:34873)
0924 10:06:27.927205 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:27.927209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:27.927228 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:06:27.927377 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:27.927381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:27.927459 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:27.927543 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:27.928863 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.929565 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.930853 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:27.931242 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:27.931335 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:27.935361 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:27.935379 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:27.935395 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:27.935430 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:27.937833 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.938544 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.938552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.938555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.938697 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:27.939042 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:27.939045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:27.939047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:27.939384 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:27.939604 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:27.939956 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 10:06:27.940127 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:06:27.941041 1251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:27.973318 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.972820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59714)
0924 10:06:27.973037 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.973132 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:06:27.974299 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:27.973950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59728)
0924 10:06:27.974113 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:27.974180 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 10:06:28.028925 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1128
I20260924 10:06:28.042171 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 768
I20260924 10:06:28.043433 830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:28.153244 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 768
I20260924 10:06:28.153290 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:28.152840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:44429 (local address 127.0.0.1:59736)
0924 10:06:28.153007 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:28.153096 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
2026-09-24T10:06:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T10:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:28Z Disabled control of system clock
I20260924 10:06:28.223328 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:41109
--webserver_interface=127.13.162.62
--webserver_port=0
--builtin_ntp_servers=127.13.162.20:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:06:28.557977 1310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:28.558269 1310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:28.558372 1310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:28.565682 1310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:28.565816 1310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:28.565867 1310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:28.565923 1310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:28.565966 1310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:28.575505 1310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:41109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:28.576803 1310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:28.578668 1310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:28.590651 1316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:28.590739 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:28.591277 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:28.591588 1310 server_base.cc:1034] running on GCE node
I20260924 10:06:28.592389 1310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:28.593933 1310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:28.595147 1310 hybrid_clock.cc:648] HybridClock initialized: now 1790244388595097 us; error 37 us; skew 500 ppm
I20260924 10:06:28.595516 1310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:28.597718 1310 webserver.cc:460] Webserver started at http://127.13.162.62:42931/ using document root <none> and password file <none>
I20260924 10:06:28.598433 1310 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:28.598534 1310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:28.598835 1310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:28.601709 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "462a1275b7964efd95f723f0cf9025dd"
format_stamp: "Formatted at 2026-09-24 10:06:28 on dist-test-slave-jvsj"
server_key: "7ea890ac4ea35b1a14ed221d90346342"
server_key_iv: "04b99cad8b87b056dc77ed3bd8c5d763"
server_key_version: "encryptionkey@0"
I20260924 10:06:28.602545 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "462a1275b7964efd95f723f0cf9025dd"
format_stamp: "Formatted at 2026-09-24 10:06:28 on dist-test-slave-jvsj"
server_key: "7ea890ac4ea35b1a14ed221d90346342"
server_key_iv: "04b99cad8b87b056dc77ed3bd8c5d763"
server_key_version: "encryptionkey@0"
I20260924 10:06:28.607745 1310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 10:06:28.611459 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:28.612950 1310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:06:28.613138 1310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "462a1275b7964efd95f723f0cf9025dd"
format_stamp: "Formatted at 2026-09-24 10:06:28 on dist-test-slave-jvsj"
server_key: "7ea890ac4ea35b1a14ed221d90346342"
server_key_iv: "04b99cad8b87b056dc77ed3bd8c5d763"
server_key_version: "encryptionkey@0"
I20260924 10:06:28.613354 1310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:28.634168 1310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:28.635119 1310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:28.635417 1310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:28.652190 1310 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:41109
I20260924 10:06:28.652216 1375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:41109 every 8 connection(s)
I20260924 10:06:28.653589 1310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:28.656785 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1310
I20260924 10:06:28.656962 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:28.657503 13960 external_mini_cluster.cc:1428] Setting key 5482ba86648971303ec70837ba1e4968
I20260924 10:06:28.657994 1376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:28.671494 1376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd: Bootstrap starting.
I20260924 10:06:28.672216 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:28.659541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.13.162.62:41109)
0924 10:06:28.660569 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:28.660580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:28.660623 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:06:28.662944 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:28.662974 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:28.663565 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:28.663982 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:28.664640 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:28.665956 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:28.667391 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:28.668104 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:28.668296 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:28.669662 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:28.669722 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:28.669738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:28.670303 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:28.670437 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:28.670463 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:28.670568 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:28.670803 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:06:28.671135 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":353,"threads_started":1}
I20260924 10:06:28.674861 1376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:28.675916 1376 log.cc:824] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:28.678787 1376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd: No bootstrap required, opened a new log
I20260924 10:06:28.686470 1376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a1275b7964efd95f723f0cf9025dd" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41109 } }
I20260924 10:06:28.686918 1376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:28.687031 1376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a1275b7964efd95f723f0cf9025dd, State: Initialized, Role: FOLLOWER
I20260924 10:06:28.687606 1376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a1275b7964efd95f723f0cf9025dd" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41109 } }
I20260924 10:06:28.687820 1376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:28.687919 1376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:28.688033 1376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:28.690099 1376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a1275b7964efd95f723f0cf9025dd" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41109 } }
I20260924 10:06:28.690608 1376 leader_election.cc:302] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 462a1275b7964efd95f723f0cf9025dd; no voters:
I20260924 10:06:28.691152 1376 leader_election.cc:288] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:28.691294 1381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:28.691987 1381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [term 1 LEADER]: Becoming Leader. State: Replica: 462a1275b7964efd95f723f0cf9025dd, State: Running, Role: LEADER
I20260924 10:06:28.692528 1381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a1275b7964efd95f723f0cf9025dd" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41109 } }
I20260924 10:06:28.693161 1376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:28.695115 1382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "462a1275b7964efd95f723f0cf9025dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a1275b7964efd95f723f0cf9025dd" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41109 } } }
I20260924 10:06:28.695566 1382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:28.696385 1388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:28.696280 1383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 462a1275b7964efd95f723f0cf9025dd. Latest consensus state: current_term: 1 leader_uuid: "462a1275b7964efd95f723f0cf9025dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a1275b7964efd95f723f0cf9025dd" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41109 } } }
I20260924 10:06:28.698966 1383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:28.704013 1388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:28.712845 1388 catalog_manager.cc:1347] Generated new cluster ID: b0427691dcf84895a3dfedb65ef21f0b
I20260924 10:06:28.712975 1388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:28.728569 1388 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:28.729940 1388 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:28.736690 1388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd: Generated new TSK 0
I20260924 10:06:28.737674 1388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:28.749770 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.1:0
--local_ip_for_outbound_sockets=127.13.162.1
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41109
--builtin_ntp_servers=127.13.162.20:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:06:29.075449 1400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:29.075745 1400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:29.075871 1400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:29.083209 1400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:29.083309 1400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:29.083448 1400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.1
I20260924 10:06:29.092723 1400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.162.1
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.1
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:29.094085 1400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:29.096027 1400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:29.108765 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:29.108693 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:29.108728 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:29.109575 1400 server_base.cc:1034] running on GCE node
I20260924 10:06:29.110095 1400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:29.111123 1400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:29.112336 1400 hybrid_clock.cc:648] HybridClock initialized: now 1790244389112267 us; error 48 us; skew 500 ppm
I20260924 10:06:29.112625 1400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:29.114777 1400 webserver.cc:460] Webserver started at http://127.13.162.1:43897/ using document root <none> and password file <none>
I20260924 10:06:29.115419 1400 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:29.115504 1400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:29.115751 1400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:29.118633 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data/instance:
uuid: "821f9bbf16d24366999160b928cc6aca"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "52efc99d022379dcde51e183b46ff9cf"
server_key_iv: "d1b5e34ce9647ce8006e402e3764926b"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.119491 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance:
uuid: "821f9bbf16d24366999160b928cc6aca"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "52efc99d022379dcde51e183b46ff9cf"
server_key_iv: "d1b5e34ce9647ce8006e402e3764926b"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.124490 1400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 10:06:29.127959 1414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:29.129340 1400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:06:29.129490 1400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal
uuid: "821f9bbf16d24366999160b928cc6aca"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "52efc99d022379dcde51e183b46ff9cf"
server_key_iv: "d1b5e34ce9647ce8006e402e3764926b"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.129679 1400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:29.144680 1400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:29.145637 1400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:29.145968 1400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:29.146993 1400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:29.148428 1400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:29.148512 1400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:29.148595 1400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:29.148648 1400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:29.178002 1400 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.1:33627
I20260924 10:06:29.178050 1526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.1:33627 every 8 connection(s)
I20260924 10:06:29.179343 1400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/data/info.pb
I20260924 10:06:29.183152 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1400
I20260924 10:06:29.183341 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-0/wal/instance
I20260924 10:06:29.183842 13960 external_mini_cluster.cc:1428] Setting key 78c5e3b7280953f6f47bcba99e45d3e5
I20260924 10:06:29.186524 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.2:0
--local_ip_for_outbound_sockets=127.13.162.2
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41109
--builtin_ntp_servers=127.13.162.20:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:06:29.191962 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:29.181829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.1:48893 (local address 127.13.162.62:41109)
0924 10:06:29.182282 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:29.182288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:29.183613 (+ 1325us) server_negotiation.cc:407] Connection header received
0924 10:06:29.185535 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:29.185540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:29.185627 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:29.185742 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:29.186950 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.187664 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.189201 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.189612 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.189731 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:29.191160 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:29.191166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:29.191171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:29.191193 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:29.191240 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:29.191255 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:29.191356 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:29.191554 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 10:06:29.191653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":206,"threads_started":1}
I20260924 10:06:29.193126 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:29.182432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:41109 (local address 127.13.162.1:48893)
0924 10:06:29.183444 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:29.183500 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:29.185255 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:29.185986 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:29.185995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:29.186088 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:29.186784 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:29.186804 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.187810 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.187814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:29.189065 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:29.189077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.189826 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.189834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:29.190061 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:29.190803 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:29.190806 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:29.190856 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:29.190949 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:29.191354 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:29.191364 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:29.191371 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:06:29.191521 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:06:29.191850 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":295,"threads_started":1}
I20260924 10:06:29.194862 1527 heartbeater.cc:347] Connected to a master server at 127.13.162.62:41109
I20260924 10:06:29.195142 1527 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:29.195873 1527 heartbeater.cc:511] Master 127.13.162.62:41109 requested a full tablet report, sending...
I20260924 10:06:29.198025 1341 ts_manager.cc:194] Registered new tserver with Master: 821f9bbf16d24366999160b928cc6aca (127.13.162.1:33627)
I20260924 10:06:29.200018 1341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.1:48893
W20260924 10:06:29.507130 1531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:29.507408 1531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:29.507505 1531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:29.514631 1531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:29.514786 1531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:29.514904 1531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.2
I20260924 10:06:29.524703 1531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.162.2
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.2
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:29.525924 1531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:29.527681 1531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:29.540560 1536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:29.540524 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:29.540486 1539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:29.541457 1531 server_base.cc:1034] running on GCE node
I20260924 10:06:29.542050 1531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:29.543049 1531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:29.544260 1531 hybrid_clock.cc:648] HybridClock initialized: now 1790244389544202 us; error 49 us; skew 500 ppm
I20260924 10:06:29.544592 1531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:29.546913 1531 webserver.cc:460] Webserver started at http://127.13.162.2:39253/ using document root <none> and password file <none>
I20260924 10:06:29.547612 1531 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:29.547715 1531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:29.548014 1531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:29.551040 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data/instance:
uuid: "a2453d936f7343bea56af34d9a8e478a"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "96b135c635cc4b8d58b2a416fb12c0ad"
server_key_iv: "5646e6b743219c81f3351081e63a06f5"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.551851 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance:
uuid: "a2453d936f7343bea56af34d9a8e478a"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "96b135c635cc4b8d58b2a416fb12c0ad"
server_key_iv: "5646e6b743219c81f3351081e63a06f5"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.557539 1531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:06:29.561533 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:29.563122 1531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:06:29.563294 1531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal
uuid: "a2453d936f7343bea56af34d9a8e478a"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "96b135c635cc4b8d58b2a416fb12c0ad"
server_key_iv: "5646e6b743219c81f3351081e63a06f5"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.563516 1531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:29.578492 1531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:29.579340 1531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:29.579620 1531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:29.580618 1531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:29.582055 1531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:29.582147 1531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:29.582254 1531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:29.582355 1531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:29.612257 1531 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.2:37223
I20260924 10:06:29.612303 1657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.2:37223 every 8 connection(s)
I20260924 10:06:29.613579 1531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/data/info.pb
I20260924 10:06:29.621654 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1531
I20260924 10:06:29.621829 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal/instance
I20260924 10:06:29.622285 13960 external_mini_cluster.cc:1428] Setting key bc9b1fec1fe661a772988e3cd138ea87
I20260924 10:06:29.625224 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.162.3:0
--local_ip_for_outbound_sockets=127.13.162.3
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41109
--builtin_ntp_servers=127.13.162.20:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:06:29.626559 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:29.616131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.2:57835 (local address 127.13.162.62:41109)
0924 10:06:29.616317 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:29.616322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:29.617806 (+ 1484us) server_negotiation.cc:407] Connection header received
0924 10:06:29.619655 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:29.619661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:29.619733 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:29.619827 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:29.620907 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.621836 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.623785 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.624298 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.624414 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:29.625850 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:29.625855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:29.625858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:29.625877 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:29.625907 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:29.625919 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:29.625989 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:29.626208 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 10:06:29.626292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:06:29.627945 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:29.616690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:41109 (local address 127.13.162.2:57835)
0924 10:06:29.617646 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:29.617690 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:29.619395 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:29.620028 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:29.620036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:29.620125 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:29.620752 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:29.620772 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.621990 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.621994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:29.623634 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:29.623648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:29.624521 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:29.624527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:29.624748 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:29.625483 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:29.625486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:29.625538 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:29.625633 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:29.626008 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:29.626017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:29.626025 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:06:29.626207 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:06:29.626648 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":314,"threads_started":1}
I20260924 10:06:29.629338 1658 heartbeater.cc:347] Connected to a master server at 127.13.162.62:41109
I20260924 10:06:29.629638 1658 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:29.630534 1658 heartbeater.cc:511] Master 127.13.162.62:41109 requested a full tablet report, sending...
I20260924 10:06:29.632098 1341 ts_manager.cc:194] Registered new tserver with Master: a2453d936f7343bea56af34d9a8e478a (127.13.162.2:37223)
I20260924 10:06:29.632915 1341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.2:57835
W20260924 10:06:29.949769 1661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:29.950073 1661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:29.950186 1661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:29.957571 1661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:29.957682 1661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:29.957825 1661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.162.3
I20260924 10:06:29.967092 1661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.162.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.162.3
--webserver_port=0
--tserver_master_addrs=127.13.162.62:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.162.3
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:29.968426 1661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:29.970171 1661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:29.982668 1669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:29.982733 1666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:29.982703 1667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:29.983453 1661 server_base.cc:1034] running on GCE node
I20260924 10:06:29.984057 1661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:29.985033 1661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:29.986246 1661 hybrid_clock.cc:648] HybridClock initialized: now 1790244389986196 us; error 45 us; skew 500 ppm
I20260924 10:06:29.986618 1661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:06:29.988915 1661 webserver.cc:460] Webserver started at http://127.13.162.3:34871/ using document root <none> and password file <none>
I20260924 10:06:29.989600 1661 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:29.989693 1661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:29.989992 1661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:29.992929 1661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data/instance:
uuid: "37ce6e6f7acc4bea812fa8c479f226bd"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "5415f4dcf837a08356f32d353b0aad89"
server_key_iv: "cbf1dc4c079c4a4b283adbe6aa8bd91b"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.993731 1661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance:
uuid: "37ce6e6f7acc4bea812fa8c479f226bd"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "5415f4dcf837a08356f32d353b0aad89"
server_key_iv: "cbf1dc4c079c4a4b283adbe6aa8bd91b"
server_key_version: "encryptionkey@0"
I20260924 10:06:29.998945 1661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:06:30.002460 1675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:30.003777 1661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:30.003926 1661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal
uuid: "37ce6e6f7acc4bea812fa8c479f226bd"
format_stamp: "Formatted at 2026-09-24 10:06:29 on dist-test-slave-jvsj"
server_key: "5415f4dcf837a08356f32d353b0aad89"
server_key_iv: "cbf1dc4c079c4a4b283adbe6aa8bd91b"
server_key_version: "encryptionkey@0"
I20260924 10:06:30.004122 1661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:30.021940 1661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:30.022853 1661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:30.023142 1661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:30.024122 1661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:06:30.025544 1661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:06:30.025641 1661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:30.025748 1661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:06:30.025817 1661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:30.055864 1661 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.3:39429
I20260924 10:06:30.055918 1787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.3:39429 every 8 connection(s)
I20260924 10:06:30.057173 1661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/data/info.pb
I20260924 10:06:30.060621 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1661
I20260924 10:06:30.060801 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal/instance
I20260924 10:06:30.061240 13960 external_mini_cluster.cc:1428] Setting key 7e3fdef6d21d8aa97cd9071f112087a3
I20260924 10:06:30.069483 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.059710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:45221 (local address 127.13.162.62:41109)
0924 10:06:30.059906 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.059911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.061394 (+ 1483us) server_negotiation.cc:407] Connection header received
0924 10:06:30.063424 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.063429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.063509 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.063589 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:30.064659 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.065393 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.067013 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.067449 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.067549 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.068819 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:30.068824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:30.068830 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:30.068847 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:30.068876 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:30.068887 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:30.068954 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.069150 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.069227 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:06:30.070474 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.060196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:41109 (local address 127.13.162.3:45221)
0924 10:06:30.061224 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.061268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:30.063177 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:30.063841 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:30.063850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:30.063941 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:06:30.064497 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.064513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.065535 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.065539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.066871 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.066883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.067636 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.067641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.067885 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.068536 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:06:30.068538 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:06:30.068576 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:06:30.068655 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:06:30.068980 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:06:30.068988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:06:30.068994 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:06:30.069134 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 10:06:30.069437 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
I20260924 10:06:30.071812 1788 heartbeater.cc:347] Connected to a master server at 127.13.162.62:41109
I20260924 10:06:30.072119 1788 heartbeater.cc:464] Registering TS with master...
I20260924 10:06:30.072860 1788 heartbeater.cc:511] Master 127.13.162.62:41109 requested a full tablet report, sending...
I20260924 10:06:30.074195 1341 ts_manager.cc:194] Registered new tserver with Master: 37ce6e6f7acc4bea812fa8c479f226bd (127.13.162.3:39429)
I20260924 10:06:30.075203 1341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.162.3:45221
I20260924 10:06:30.076303 13960 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:06:30.085421 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.080026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.13.162.62:41109)
0924 10:06:30.080207 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.080212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.080590 (+ 378us) server_negotiation.cc:407] Connection header received
0924 10:06:30.080719 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.080723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.080817 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.080897 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:30.081343 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.082054 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.083401 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.083800 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.083890 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.084868 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:30.084873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:30.084876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:30.084896 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:30.084928 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:30.084939 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:30.085006 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.085099 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.085178 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:06:30.089293 1341 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:06:30.096248 1341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:06:30.118829 1800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.108038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.3:39429 (local address 127.0.0.1:48992)
0924 10:06:30.108915 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.108962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:30.109159 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:30.110367 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:30.110375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:30.110417 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:30.110809 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.110821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.113177 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.113183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.115713 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.115735 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.117756 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.117764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.118045 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.118098 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 10:06:30.118210 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:06:30.118424 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":27,"thread_start_us":866,"threads_started":1}
I20260924 10:06:30.120453 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.108632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.13.162.3:39429)
0924 10:06:30.109712 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.109722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.109756 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:06:30.109850 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.109857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.110101 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.111007 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:30.111281 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.113015 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.115962 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.117529 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.119512 (+ 1983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.119838 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.119911 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.120074 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":879,"threads_started":1}
I20260924 10:06:30.121240 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.108722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.13.162.1:33627)
0924 10:06:30.109685 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.109693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.110021 (+ 328us) server_negotiation.cc:407] Connection header received
0924 10:06:30.110134 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.110142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.110414 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.110597 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:30.112564 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.114153 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.118638 (+ 4485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.120048 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.120152 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.120358 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.120748 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.120915 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":915,"threads_started":1}
I20260924 10:06:30.122051 1801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.108217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:39956)
0924 10:06:30.109777 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.109792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:30.109907 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:30.110830 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:30.110835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:30.110866 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:30.112350 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.112364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.114351 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.114357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.118412 (+ 4055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.118425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.120263 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.120269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.120474 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.120554 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 10:06:30.121540 (+ 986us) client_negotiation.cc:240] Negotiation successful
0924 10:06:30.121677 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":360,"thread_start_us":863,"threads_started":1}
I20260924 10:06:30.124269 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 6873b02960a34c64b8eaaf8787c8c373 (DEFAULT_TABLE table=test-table [id=10576285e26a42179718bdb5501e83aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:30.125276 1462 tablet_service.cc:1467] Processing CreateTablet for tablet 6873b02960a34c64b8eaaf8787c8c373 (DEFAULT_TABLE table=test-table [id=10576285e26a42179718bdb5501e83aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:30.125878 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6873b02960a34c64b8eaaf8787c8c373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:30.126493 1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6873b02960a34c64b8eaaf8787c8c373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:30.126503 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.109895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:37223 (local address 127.0.0.1:60616)
0924 10:06:30.118999 (+ 9104us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.119017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:30.119129 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:30.119899 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:30.119903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:30.119927 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:30.120257 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.120267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.121977 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.121982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.124019 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.124035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.125756 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.125763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.125947 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.125980 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:06:30.126067 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:06:30.126191 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8984,"thread_start_us":8682,"threads_started":1}
I20260924 10:06:30.126662 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.110052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.13.162.2:37223)
0924 10:06:30.118413 (+ 8361us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.118423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.119253 (+ 830us) server_negotiation.cc:407] Connection header received
0924 10:06:30.119369 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.119378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.119709 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.119881 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:30.120429 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.121820 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.124242 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.125566 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.125695 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.125930 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.126137 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.126311 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8128,"thread_start_us":7991,"threads_started":1}
I20260924 10:06:30.130286 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 6873b02960a34c64b8eaaf8787c8c373 (DEFAULT_TABLE table=test-table [id=10576285e26a42179718bdb5501e83aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:06:30.131645 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6873b02960a34c64b8eaaf8787c8c373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:30.138938 1806 tablet_bootstrap.cc:492] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: Bootstrap starting.
I20260924 10:06:30.140928 1807 tablet_bootstrap.cc:492] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Bootstrap starting.
I20260924 10:06:30.142912 1806 tablet_bootstrap.cc:654] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:30.144131 1806 log.cc:824] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:30.144342 1807 tablet_bootstrap.cc:654] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:30.144775 1808 tablet_bootstrap.cc:492] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Bootstrap starting.
I20260924 10:06:30.145570 1807 log.cc:824] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:30.148142 1808 tablet_bootstrap.cc:654] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:30.148664 1806 tablet_bootstrap.cc:492] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: No bootstrap required, opened a new log
I20260924 10:06:30.148941 1806 ts_tablet_manager.cc:1397] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 10:06:30.149206 1807 tablet_bootstrap.cc:492] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: No bootstrap required, opened a new log
I20260924 10:06:30.149307 1808 log.cc:824] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:30.149452 1807 ts_tablet_manager.cc:1397] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 10:06:30.152163 1808 tablet_bootstrap.cc:492] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: No bootstrap required, opened a new log
I20260924 10:06:30.152410 1808 ts_tablet_manager.cc:1397] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 10:06:30.156977 1807 raft_consensus.cc:348] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.157487 1807 raft_consensus.cc:374] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:30.157608 1807 raft_consensus.cc:729] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ce6e6f7acc4bea812fa8c479f226bd, State: Initialized, Role: FOLLOWER
I20260924 10:06:30.158257 1807 consensus_queue.cc:260] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.159320 1788 heartbeater.cc:503] Master 127.13.162.62:41109 was elected leader, sending a full tablet report...
I20260924 10:06:30.159349 1806 raft_consensus.cc:348] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.159775 1806 raft_consensus.cc:374] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:30.159929 1806 raft_consensus.cc:729] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821f9bbf16d24366999160b928cc6aca, State: Initialized, Role: FOLLOWER
I20260924 10:06:30.160663 1807 ts_tablet_manager.cc:1428] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 10:06:30.160549 1806 consensus_queue.cc:260] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.161720 1527 heartbeater.cc:503] Master 127.13.162.62:41109 was elected leader, sending a full tablet report...
I20260924 10:06:30.162343 1806 ts_tablet_manager.cc:1428] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260924 10:06:30.162292 1808 raft_consensus.cc:348] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.162734 1808 raft_consensus.cc:374] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:30.162825 1808 raft_consensus.cc:729] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2453d936f7343bea56af34d9a8e478a, State: Initialized, Role: FOLLOWER
I20260924 10:06:30.163354 1808 consensus_queue.cc:260] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.164309 1658 heartbeater.cc:503] Master 127.13.162.62:41109 was elected leader, sending a full tablet report...
I20260924 10:06:30.166213 1808 ts_tablet_manager.cc:1428] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260924 10:06:30.181471 1528 tablet.cc:2369] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:06:30.309047 1789 tablet.cc:2369] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:30.354214 1812 raft_consensus.cc:482] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:06:30.354547 1812 raft_consensus.cc:504] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.356194 1812 leader_election.cc:288] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 821f9bbf16d24366999160b928cc6aca (127.13.162.1:33627), a2453d936f7343bea56af34d9a8e478a (127.13.162.2:37223)
I20260924 10:06:30.361878 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.356569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.13.162.3:34035)
0924 10:06:30.356783 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.356813 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:30.356960 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:30.357344 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:30.357348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:30.357376 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:30.357574 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.357586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.358629 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.358634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.360263 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.360278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.361317 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.361324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.361465 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.361489 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:06:30.361555 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:06:30.361647 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":3}
I20260924 10:06:30.362016 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.356621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:34035 (local address 127.13.162.1:33627)
0924 10:06:30.356820 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.356826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.356876 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:06:30.357140 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.357144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.357212 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.357292 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:30.357761 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.358484 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.360420 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.361178 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.361300 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.361368 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.361599 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.361729 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:06:30.363204 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.357301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.162.3:45445 (local address 127.13.162.2:37223)
0924 10:06:30.357506 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.357512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.357973 (+ 461us) server_negotiation.cc:407] Connection header received
0924 10:06:30.358112 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.358117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.358174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.358245 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:06:30.358826 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.359509 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.361389 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.362462 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.362599 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.362646 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.362947 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.363036 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:06:30.363107 1482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6873b02960a34c64b8eaaf8787c8c373" candidate_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "821f9bbf16d24366999160b928cc6aca" is_pre_election: true
I20260924 10:06:30.363358 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.357012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.2:37223 (local address 127.13.162.3:45445)
0924 10:06:30.357880 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.357894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:06:30.357997 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:06:30.358269 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:06:30.358272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:06:30.358294 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:06:30.358654 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.358664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.359664 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.359669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.361191 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:06:30.361203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.362632 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.362639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:06:30.362801 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.362835 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:06:30.362924 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:06:30.363042 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":23,"thread_start_us":570,"threads_started":1}
I20260924 10:06:30.363716 1482 raft_consensus.cc:2454] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ce6e6f7acc4bea812fa8c479f226bd in term 0.
I20260924 10:06:30.364264 1613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6873b02960a34c64b8eaaf8787c8c373" candidate_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2453d936f7343bea56af34d9a8e478a" is_pre_election: true
I20260924 10:06:30.364382 1678 leader_election.cc:302] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ce6e6f7acc4bea812fa8c479f226bd, 821f9bbf16d24366999160b928cc6aca; no voters:
I20260924 10:06:30.364813 1812 raft_consensus.cc:2781] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:06:30.364805 1613 raft_consensus.cc:2454] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ce6e6f7acc4bea812fa8c479f226bd in term 0.
I20260924 10:06:30.364965 1812 raft_consensus.cc:482] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:06:30.365058 1812 raft_consensus.cc:3037] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 0 FOLLOWER]: Advancing to term 1
W20260924 10:06:30.365612 1659 tablet.cc:2369] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:06:30.367758 1812 raft_consensus.cc:504] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.368314 1812 leader_election.cc:288] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [CANDIDATE]: Term 1 election: Requested vote from peers 821f9bbf16d24366999160b928cc6aca (127.13.162.1:33627), a2453d936f7343bea56af34d9a8e478a (127.13.162.2:37223)
I20260924 10:06:30.368840 1482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6873b02960a34c64b8eaaf8787c8c373" candidate_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "821f9bbf16d24366999160b928cc6aca"
I20260924 10:06:30.368839 1613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6873b02960a34c64b8eaaf8787c8c373" candidate_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2453d936f7343bea56af34d9a8e478a"
I20260924 10:06:30.369051 1613 raft_consensus.cc:3037] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:30.369051 1482 raft_consensus.cc:3037] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:30.372164 1482 raft_consensus.cc:2454] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ce6e6f7acc4bea812fa8c479f226bd in term 1.
I20260924 10:06:30.372166 1613 raft_consensus.cc:2454] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ce6e6f7acc4bea812fa8c479f226bd in term 1.
I20260924 10:06:30.372582 1678 leader_election.cc:302] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ce6e6f7acc4bea812fa8c479f226bd, 821f9bbf16d24366999160b928cc6aca; no voters:
I20260924 10:06:30.372864 1812 raft_consensus.cc:2781] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:30.373265 1812 raft_consensus.cc:686] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 1 LEADER]: Becoming Leader. State: Replica: 37ce6e6f7acc4bea812fa8c479f226bd, State: Running, Role: LEADER
I20260924 10:06:30.373806 1812 consensus_queue.cc:237] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } }
I20260924 10:06:30.378423 1340 catalog_manager.cc:5495] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ce6e6f7acc4bea812fa8c479f226bd (127.13.162.3). New cstate: current_term: 1 leader_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ce6e6f7acc4bea812fa8c479f226bd" member_type: VOTER last_known_addr { host: "127.13.162.3" port: 39429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 } health_report { overall_health: UNKNOWN } } }
I20260924 10:06:30.413230 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.407896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.13.162.3:39429)
0924 10:06:30.408106 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.408111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.408134 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:06:30.408384 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.408388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.408478 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.408606 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:06:30.409012 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.409741 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.411119 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.411539 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.411647 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.412126 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:06:30.412662 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:06:30.412798 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.412903 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.412985 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:06:30.421837 1613 raft_consensus.cc:1261] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 1 FOLLOWER]: Refusing update from remote peer 37ce6e6f7acc4bea812fa8c479f226bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:30.421890 1482 raft_consensus.cc:1261] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 1 FOLLOWER]: Refusing update from remote peer 37ce6e6f7acc4bea812fa8c479f226bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:06:30.422791 1816 consensus_queue.cc:1035] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "821f9bbf16d24366999160b928cc6aca" member_type: VOTER last_known_addr { host: "127.13.162.1" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:30.423170 1812 consensus_queue.cc:1035] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2453d936f7343bea56af34d9a8e478a" member_type: VOTER last_known_addr { host: "127.13.162.2" port: 37223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:06:30.453474 1340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:30.453816 1340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:06:30.457880 1340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 462a1275b7964efd95f723f0cf9025dd: Sending DeleteTablet for 3 replicas of tablet 6873b02960a34c64b8eaaf8787c8c373
I20260924 10:06:30.459091 1723 tablet_service.cc:1514] Processing DeleteTablet for tablet 6873b02960a34c64b8eaaf8787c8c373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:30 UTC) from {username='slave'} at 127.0.0.1:48992
I20260924 10:06:30.459306 1462 tablet_service.cc:1514] Processing DeleteTablet for tablet 6873b02960a34c64b8eaaf8787c8c373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:30 UTC) from {username='slave'} at 127.0.0.1:39956
I20260924 10:06:30.459297 1593 tablet_service.cc:1514] Processing DeleteTablet for tablet 6873b02960a34c64b8eaaf8787c8c373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:06:30 UTC) from {username='slave'} at 127.0.0.1:60616
I20260924 10:06:30.469367 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1400
I20260924 10:06:30.472137 1829 tablet_replica.cc:330] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: stopping tablet replica
I20260924 10:06:30.472736 1829 raft_consensus.cc:2229] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:06:30.473594 1829 raft_consensus.cc:2258] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:30.475283 1830 tablet_replica.cc:330] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: stopping tablet replica
I20260924 10:06:30.475776 1830 raft_consensus.cc:2229] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:30.476336 1830 raft_consensus.cc:2258] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:30.478813 1830 ts_tablet_manager.cc:1905] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:30.483382 1831 tablet_replica.cc:330] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca: stopping tablet replica
I20260924 10:06:30.484004 1831 raft_consensus.cc:2229] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:06:30.484119 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.470850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.13.162.1:33627)
0924 10:06:30.471060 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.471067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.474546 (+ 3479us) server_negotiation.cc:407] Connection header received
0924 10:06:30.474722 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.474728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.474898 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.475003 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:30.475655 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.477096 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.479157 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.480082 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.480204 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.481839 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:30.481853 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:30.481872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:30.482777 (+ 905us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:30.482845 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:30.482872 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:30.482984 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.483609 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.483739 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:06:30.485054 1522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:30.486084 1830 ts_tablet_manager.cc:1918] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:30.486304 1830 log.cc:1196] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-1/wal/wals/6873b02960a34c64b8eaaf8787c8c373
I20260924 10:06:30.486404 1829 ts_tablet_manager.cc:1905] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:06:30.487051 1830 ts_tablet_manager.cc:1939] T 6873b02960a34c64b8eaaf8787c8c373 P a2453d936f7343bea56af34d9a8e478a: Deleting consensus metadata
I20260924 10:06:30.489022 1327 catalog_manager.cc:4841] TS a2453d936f7343bea56af34d9a8e478a (127.13.162.2:37223): tablet 6873b02960a34c64b8eaaf8787c8c373 (table test-table [id=10576285e26a42179718bdb5501e83aa]) successfully deleted
I20260924 10:06:30.493748 1829 ts_tablet_manager.cc:1918] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:06:30.493917 1829 log.cc:1196] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Deleting WAL directory at /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244085964582-13960-0/minicluster-data/ts-2/wal/wals/6873b02960a34c64b8eaaf8787c8c373
I20260924 10:06:30.494479 1829 ts_tablet_manager.cc:1939] T 6873b02960a34c64b8eaaf8787c8c373 P 37ce6e6f7acc4bea812fa8c479f226bd: Deleting consensus metadata
I20260924 10:06:30.496001 1326 catalog_manager.cc:4841] TS 37ce6e6f7acc4bea812fa8c479f226bd (127.13.162.3:39429): tablet 6873b02960a34c64b8eaaf8787c8c373 (table test-table [id=10576285e26a42179718bdb5501e83aa]) successfully deleted
I20260924 10:06:30.638581 1831 raft_consensus.cc:2258] T 6873b02960a34c64b8eaaf8787c8c373 P 821f9bbf16d24366999160b928cc6aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:06:30.639391 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1400
I20260924 10:06:30.661638 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1531
W20260924 10:06:30.661859 1327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.162.1:33627 (error 108)
I20260924 10:06:30.674165 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.671392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:39982)
0924 10:06:30.673878 (+ 2486us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.673999 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2364,"thread_start_us":2257,"threads_started":1}
W20260924 10:06:30.674515 1327 catalog_manager.cc:4584] TS 821f9bbf16d24366999160b928cc6aca (127.13.162.1:33627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6873b02960a34c64b8eaaf8787c8c373: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
I20260924 10:06:30.675135 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.662570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.13.162.2:37223)
0924 10:06:30.662763 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.662770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.662898 (+ 128us) server_negotiation.cc:407] Connection header received
0924 10:06:30.663076 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.663082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.663210 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.663332 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:30.663952 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.665061 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.667012 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.667645 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.667764 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.673359 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:30.673372 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:30.673391 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:30.674179 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:30.674257 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:30.674295 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:30.674429 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.674682 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.674817 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:06:30.675870 1653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:30.728355 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.727831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:39998)
0924 10:06:30.728070 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.728178 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 10:06:30.729598 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.729125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:40002)
0924 10:06:30.729317 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.729438 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:06:30.764343 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1531
I20260924 10:06:30.778201 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1661
I20260924 10:06:30.785287 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.779043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.13.162.3:39429)
0924 10:06:30.779218 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:30.779225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:30.779362 (+ 137us) server_negotiation.cc:407] Connection header received
0924 10:06:30.779546 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:30.779550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:30.779653 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:30.779758 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:30.780199 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.780974 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.782179 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:30.782651 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:30.782768 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:30.783911 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:30.783921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:30.783936 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:30.784543 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:30.784598 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:30.784619 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:30.784712 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:30.784858 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 10:06:30.784953 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:06:30.785904 1783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:30.790964 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.790481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:40016)
0924 10:06:30.790684 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.790794 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 10:06:30.792045 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.791635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:40018)
0924 10:06:30.791803 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.791890 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:06:30.872766 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.872156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:40024)
0924 10:06:30.872468 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.872584 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 10:06:30.873867 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:30.873419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.1:33627 (local address 127.0.0.1:40040)
0924 10:06:30.873629 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 10:06:30.873712 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.1:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 10:06:30.905722 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1661
I20260924 10:06:30.921109 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1310
I20260924 10:06:30.922201 1371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:31.029800 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1310
2026-09-24T10:06:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2875 ms)
[----------] 6 tests from EncryptionPolicyTest (21051 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:31 dist-test-slave-jvsj krb5kdc[1841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:31 dist-test-slave-jvsj krb5kdc[1841](info): set up 2 sockets
Sep 24 10:06:31 dist-test-slave-jvsj krb5kdc[1841](info): commencing operation
krb5kdc: starting...
W20260924 10:06:32.050801 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:32 dist-test-slave-jvsj krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:32.291971 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:32.613579 1854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:32.613876 1854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:32.614006 1854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:32.621322 1854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:32.621439 1854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:32.621510 1854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:32.621592 1854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:32.621645 1854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:32.631688 1854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:32.633073 1854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:32.634410 1854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:32.645751 1860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:32.645867 1862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:32.645774 1859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:32.646498 1854 server_base.cc:1034] running on GCE node
I20260924 10:06:32.647264 1854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:32.648670 1854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:32.649866 1854 hybrid_clock.cc:648] HybridClock initialized: now 1790244392649812 us; error 44 us; skew 500 ppm
I20260924 10:06:32.650197 1854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:32 dist-test-slave-jvsj krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:32.653151 1854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:32.654577 1854 webserver.cc:460] Webserver started at http://127.0.0.1:46775/ using document root <none> and password file <none>
I20260924 10:06:32.655226 1854 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:32.655328 1854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:32.655629 1854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:32.658599 1854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "d9e9d503e76648179004d2ec48ef8e05"
format_stamp: "Formatted at 2026-09-24 10:06:32 on dist-test-slave-jvsj"
server_key: "b7589d3cfb2a1fe3e66c61456eb94bcb"
server_key_iv: "192decb2e902e707b6a969da907851ad"
server_key_version: "encryptionkey@0"
I20260924 10:06:32.659353 1854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "d9e9d503e76648179004d2ec48ef8e05"
format_stamp: "Formatted at 2026-09-24 10:06:32 on dist-test-slave-jvsj"
server_key: "b7589d3cfb2a1fe3e66c61456eb94bcb"
server_key_iv: "192decb2e902e707b6a969da907851ad"
server_key_version: "encryptionkey@0"
I20260924 10:06:32.664494 1854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:32.668076 1869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:32.669445 1854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:06:32.669603 1854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "d9e9d503e76648179004d2ec48ef8e05"
format_stamp: "Formatted at 2026-09-24 10:06:32 on dist-test-slave-jvsj"
server_key: "b7589d3cfb2a1fe3e66c61456eb94bcb"
server_key_iv: "192decb2e902e707b6a969da907851ad"
server_key_version: "encryptionkey@0"
I20260924 10:06:32.669787 1854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:32.687855 1854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:32.691814 1854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:32.692076 1854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:32.709175 1854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43519
I20260924 10:06:32.709196 1920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43519 every 8 connection(s)
I20260924 10:06:32.710582 1854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:32.714159 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1854
I20260924 10:06:32.714255 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:32.714371 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:32.714968 13960 external_mini_cluster.cc:1428] Setting key 9d72b716d10035c9cc464b6f449361e1
I20260924 10:06:32.727826 1921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05: Bootstrap starting.
Sep 24 10:06:32 dist-test-slave-jvsj krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:06:32.731617 1921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:32.732795 1921 log.cc:824] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:32.736089 1921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05: No bootstrap required, opened a new log
I20260924 10:06:32.738866 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:32.716751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51425 (local address 127.0.0.1:43519)
0924 10:06:32.717557 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:32.717569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:32.717612 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:06:32.719956 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:32.719982 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:32.720440 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:32.720799 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:32.722523 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:32.723853 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:32.725628 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:32.726310 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:32.726490 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:32.731190 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:32.731197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:32.731213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:32.731250 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:32.734586 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:32.735477 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:32.735483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:32.735489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:32.735646 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:32.736123 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:32.736126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:32.736128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:32.736681 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:32.736946 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:32.737381 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 10:06:32.737650 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":359,"threads_started":1}
I20260924 10:06:32.742779 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:32.744218 1921 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e9d503e76648179004d2ec48ef8e05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260924 10:06:32.744544 1921 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:32.744638 1921 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e9d503e76648179004d2ec48ef8e05, State: Initialized, Role: FOLLOWER
I20260924 10:06:32.745153 1921 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e9d503e76648179004d2ec48ef8e05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260924 10:06:32.745368 1921 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:32.745505 1921 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:32.745679 1921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:32.748191 1921 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e9d503e76648179004d2ec48ef8e05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260924 10:06:32.748962 1921 leader_election.cc:302] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9e9d503e76648179004d2ec48ef8e05; no voters:
I20260924 10:06:32.749522 1921 leader_election.cc:288] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:32.749671 1931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:32.751662 1921 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:32.756853 1931 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [term 1 LEADER]: Becoming Leader. State: Replica: d9e9d503e76648179004d2ec48ef8e05, State: Running, Role: LEADER
I20260924 10:06:32.757638 1931 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e9d503e76648179004d2ec48ef8e05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260924 10:06:32.760493 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9e9d503e76648179004d2ec48ef8e05. Latest consensus state: current_term: 1 leader_uuid: "d9e9d503e76648179004d2ec48ef8e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e9d503e76648179004d2ec48ef8e05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } } }
I20260924 10:06:32.760807 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:32.761309 1932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9e9d503e76648179004d2ec48ef8e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e9d503e76648179004d2ec48ef8e05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } } }
I20260924 10:06:32.761595 1932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:32.765372 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:32.747416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54555 (local address 127.0.0.1:43519)
0924 10:06:32.747602 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:32.747609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:32.747961 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:06:32.748100 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:32.748105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:32.748188 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:32.748295 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:32.749830 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:32.751013 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:32.752966 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:32.753682 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:32.753811 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:32.757355 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:32.757362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:32.757366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:32.757405 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:32.760998 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:32.763089 (+ 2091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:32.763093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:32.763095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:32.763223 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:32.763826 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:32.763832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:32.763835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:32.764244 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:32.764474 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:32.764908 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 10:06:32.765043 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:06:32.768188 1946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:06:32.768296 1946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:06:32.768414 1945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:32.769840 1945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:32.778620 1945 catalog_manager.cc:1347] Generated new cluster ID: c7340b01f6094af4b8556f79d0e3b48c
I20260924 10:06:32.778735 1945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:32.797515 1945 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:32.799360 1945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:32.810083 1945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9e9d503e76648179004d2ec48ef8e05: Generated new TSK 0
I20260924 10:06:32.811179 1945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:32.823868 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1854
I20260924 10:06:32.825271 1916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:32.918067 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1854
W20260924 10:06:32.932219 1839 connection.cc:535] client connection to 127.0.0.1:43519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43519): Connection reset by peer (error 104)
2026-09-24T10:06:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:33 dist-test-slave-jvsj krb5kdc[1953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:33 dist-test-slave-jvsj krb5kdc[1953](info): set up 2 sockets
Sep 24 10:06:33 dist-test-slave-jvsj krb5kdc[1953](info): commencing operation
krb5kdc: starting...
W20260924 10:06:33.926473 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:34 dist-test-slave-jvsj krb5kdc[1953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:34.157475 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:34.474381 1966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:34.474707 1966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:34.474850 1966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:34.482142 1966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:34.482264 1966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:34.482359 1966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:34.482442 1966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:34.482503 1966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:34.492403 1966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:34.493722 1966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:34.495055 1966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:34.506250 1974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:34.506227 1972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:34.506189 1971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:34.507090 1966 server_base.cc:1034] running on GCE node
I20260924 10:06:34.507838 1966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:34.509200 1966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:34.510418 1966 hybrid_clock.cc:648] HybridClock initialized: now 1790244394510352 us; error 55 us; skew 500 ppm
I20260924 10:06:34.510756 1966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:34 dist-test-slave-jvsj krb5kdc[1953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:34.513705 1966 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:34.515147 1966 webserver.cc:460] Webserver started at http://127.0.0.1:41195/ using document root <none> and password file <none>
I20260924 10:06:34.515786 1966 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:34.515900 1966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:34.516180 1966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:34.519200 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "77999346d3b24d20bc7bb4d7c6824799"
format_stamp: "Formatted at 2026-09-24 10:06:34 on dist-test-slave-jvsj"
server_key: "7d00cfc2138c3c790ae2ef8ea8eb7eff"
server_key_iv: "9b9375584e1f8541c5e33658dec2a7a7"
server_key_version: "encryptionkey@0"
I20260924 10:06:34.519977 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "77999346d3b24d20bc7bb4d7c6824799"
format_stamp: "Formatted at 2026-09-24 10:06:34 on dist-test-slave-jvsj"
server_key: "7d00cfc2138c3c790ae2ef8ea8eb7eff"
server_key_iv: "9b9375584e1f8541c5e33658dec2a7a7"
server_key_version: "encryptionkey@0"
I20260924 10:06:34.525298 1966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:06:34.528780 1981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:34.530099 1966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:06:34.530252 1966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "77999346d3b24d20bc7bb4d7c6824799"
format_stamp: "Formatted at 2026-09-24 10:06:34 on dist-test-slave-jvsj"
server_key: "7d00cfc2138c3c790ae2ef8ea8eb7eff"
server_key_iv: "9b9375584e1f8541c5e33658dec2a7a7"
server_key_version: "encryptionkey@0"
I20260924 10:06:34.530472 1966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:34.566244 1966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:34.570061 1966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:34.570336 1966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:34.587066 1966 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45195
I20260924 10:06:34.587095 2032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45195 every 8 connection(s)
I20260924 10:06:34.588387 1966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:34.589116 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 1966
I20260924 10:06:34.589274 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:34.589825 13960 external_mini_cluster.cc:1428] Setting key 572ae5e839a6165320c8c5a482c154d5
I20260924 10:06:34.592798 2033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:34.602208 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:34.591621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38765 (local address 127.0.0.1:45195)
0924 10:06:34.592350 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:34.592362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:34.592402 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:06:34.594099 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:34.594121 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:34.594567 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:34.594913 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:34.595468 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:34.596777 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:34.598048 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:34.598745 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:34.598903 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:34.600026 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:34.600033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:34.600046 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:34.600586 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:34.600657 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:34.600684 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:34.600837 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:34.601115 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:06:34.601273 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
I20260924 10:06:34.603608 2033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799: Bootstrap starting.
I20260924 10:06:34.605901 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:34.606875 2033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:34.607795 2033 log.cc:824] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:34.610857 2033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799: No bootstrap required, opened a new log
I20260924 10:06:34.615737 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:34.610081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55113 (local address 127.0.0.1:45195)
0924 10:06:34.610212 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:34.610218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:34.610651 (+ 433us) server_negotiation.cc:407] Connection header received
0924 10:06:34.610828 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:34.610832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:34.610890 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:34.610975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:34.611432 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:34.612180 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:34.613408 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:34.613858 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:34.613982 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:34.615025 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:34.615033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:34.615036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:34.615056 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:34.615093 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:34.615107 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:34.615207 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:34.615423 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 10:06:34.615480 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:06:34.618405 2033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77999346d3b24d20bc7bb4d7c6824799" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260924 10:06:34.618755 2033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:34.618842 2033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77999346d3b24d20bc7bb4d7c6824799, State: Initialized, Role: FOLLOWER
I20260924 10:06:34.619343 2033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77999346d3b24d20bc7bb4d7c6824799" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260924 10:06:34.619560 2033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:34.619673 2033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:34.619822 2033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:34.622152 2033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77999346d3b24d20bc7bb4d7c6824799" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260924 10:06:34.622713 2033 leader_election.cc:302] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77999346d3b24d20bc7bb4d7c6824799; no voters:
I20260924 10:06:34.623221 2033 leader_election.cc:288] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:34.623363 2043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:34.624014 2043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [term 1 LEADER]: Becoming Leader. State: Replica: 77999346d3b24d20bc7bb4d7c6824799, State: Running, Role: LEADER
I20260924 10:06:34.624491 2043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77999346d3b24d20bc7bb4d7c6824799" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260924 10:06:34.625146 2033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:34.627072 2045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77999346d3b24d20bc7bb4d7c6824799. Latest consensus state: current_term: 1 leader_uuid: "77999346d3b24d20bc7bb4d7c6824799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77999346d3b24d20bc7bb4d7c6824799" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } } }
I20260924 10:06:34.627450 2045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:34.628147 2044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77999346d3b24d20bc7bb4d7c6824799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77999346d3b24d20bc7bb4d7c6824799" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } } }
I20260924 10:06:34.628438 2044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:34.630422 2050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:34.635370 2050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:34.644624 2050 catalog_manager.cc:1347] Generated new cluster ID: abdf90656e524d1498029ccd8ace03cb
I20260924 10:06:34.644729 2050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:34.660440 2050 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:34.661681 2050 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:34.670941 2050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77999346d3b24d20bc7bb4d7c6824799: Generated new TSK 0
I20260924 10:06:34.671754 2050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:34.677146 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1966
W20260924 10:06:34.678264 2028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38765
W20260924 10:06:34.679045 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 1966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:34.679234 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 1966
W20260924 10:06:34.692837 1951 connection.cc:535] client connection to 127.0.0.1:45195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45195): Connection reset by peer (error 104)
2026-09-24T10:06:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:34 dist-test-slave-jvsj krb5kdc[2064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:34 dist-test-slave-jvsj krb5kdc[2064](info): set up 2 sockets
Sep 24 10:06:34 dist-test-slave-jvsj krb5kdc[2064](info): commencing operation
krb5kdc: starting...
W20260924 10:06:35.698521 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.012s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:35 dist-test-slave-jvsj krb5kdc[2064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:35.937616 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:36.262372 2077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:36.262689 2077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:36.262831 2077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:36.270224 2077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:36.270378 2077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:36.270460 2077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:36.270547 2077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:36.270603 2077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:36.281026 2077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:36.282459 2077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:36.283831 2077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:36.295781 2083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:36.295809 2082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:36.296391 2085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:36.296904 2077 server_base.cc:1034] running on GCE node
I20260924 10:06:36.297894 2077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:36.299440 2077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:36.300665 2077 hybrid_clock.cc:648] HybridClock initialized: now 1790244396300602 us; error 56 us; skew 500 ppm
I20260924 10:06:36.301017 2077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:36 dist-test-slave-jvsj krb5kdc[2064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:36.310408 2077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:36.312069 2077 webserver.cc:460] Webserver started at http://127.0.0.1:40967/ using document root <none> and password file <none>
I20260924 10:06:36.312820 2077 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:36.312945 2077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:36.313251 2077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:36.316396 2077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "5093edb0975c4ca382fd6c7bac31237f"
format_stamp: "Formatted at 2026-09-24 10:06:36 on dist-test-slave-jvsj"
server_key: "067b87075a99ec2fcd5684efe1152c2d"
server_key_iv: "5cf4ec2609f6b3b1c5990af032c2aecd"
server_key_version: "encryptionkey@0"
I20260924 10:06:36.317219 2077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "5093edb0975c4ca382fd6c7bac31237f"
format_stamp: "Formatted at 2026-09-24 10:06:36 on dist-test-slave-jvsj"
server_key: "067b87075a99ec2fcd5684efe1152c2d"
server_key_iv: "5cf4ec2609f6b3b1c5990af032c2aecd"
server_key_version: "encryptionkey@0"
I20260924 10:06:36.322654 2077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:06:36.326208 2092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:36.327648 2077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:06:36.327804 2077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "5093edb0975c4ca382fd6c7bac31237f"
format_stamp: "Formatted at 2026-09-24 10:06:36 on dist-test-slave-jvsj"
server_key: "067b87075a99ec2fcd5684efe1152c2d"
server_key_iv: "5cf4ec2609f6b3b1c5990af032c2aecd"
server_key_version: "encryptionkey@0"
I20260924 10:06:36.327998 2077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:36.333562 2077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:36.337604 2077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:36.337869 2077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:36.354874 2077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46085
I20260924 10:06:36.354905 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46085 every 8 connection(s)
I20260924 10:06:36.356244 2077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:36.359980 2144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:36.360477 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2077
I20260924 10:06:36.360662 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:36.361187 13960 external_mini_cluster.cc:1428] Setting key 2c51ad2d70b3c605e77caec5cb3f0607
I20260924 10:06:36.369513 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:36.363111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37093 (local address 127.0.0.1:46085)
0924 10:06:36.363929 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:36.363942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:36.363986 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:06:36.366088 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:36.366112 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:36.366521 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:36.366871 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:36.367906 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:36.367912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:36.367925 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:36.367939 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:36.368029 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:36.368054 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:36.368214 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:36.368307 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 10:06:36.368475 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":385,"threads_started":1}
I20260924 10:06:36.373333 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:36.373500 2144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f: Bootstrap starting.
I20260924 10:06:36.376523 2144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:36.377657 2144 log.cc:824] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:36.379691 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:36.377530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59387 (local address 127.0.0.1:46085)
0924 10:06:36.377711 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:36.377717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:36.378008 (+ 291us) server_negotiation.cc:407] Connection header received
0924 10:06:36.378169 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:36.378174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:36.378236 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:36.378374 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:36.379136 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:36.379142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:36.379146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:36.379163 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:36.379206 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:36.379225 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:36.379332 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:36.379375 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 10:06:36.379426 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:06:36.380806 2144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f: No bootstrap required, opened a new log
I20260924 10:06:36.388298 2144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5093edb0975c4ca382fd6c7bac31237f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46085 } }
I20260924 10:06:36.388685 2144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:36.388787 2144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5093edb0975c4ca382fd6c7bac31237f, State: Initialized, Role: FOLLOWER
I20260924 10:06:36.389374 2144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5093edb0975c4ca382fd6c7bac31237f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46085 } }
I20260924 10:06:36.389588 2144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:36.389727 2144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:36.389871 2144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:36.392432 2144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5093edb0975c4ca382fd6c7bac31237f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46085 } }
I20260924 10:06:36.393029 2144 leader_election.cc:302] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5093edb0975c4ca382fd6c7bac31237f; no voters:
I20260924 10:06:36.393610 2144 leader_election.cc:288] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:36.393754 2154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:36.394721 2154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [term 1 LEADER]: Becoming Leader. State: Replica: 5093edb0975c4ca382fd6c7bac31237f, State: Running, Role: LEADER
I20260924 10:06:36.395300 2154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5093edb0975c4ca382fd6c7bac31237f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46085 } }
I20260924 10:06:36.395709 2144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:36.398502 2155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5093edb0975c4ca382fd6c7bac31237f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5093edb0975c4ca382fd6c7bac31237f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46085 } } }
I20260924 10:06:36.398928 2155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:36.398433 2156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5093edb0975c4ca382fd6c7bac31237f. Latest consensus state: current_term: 1 leader_uuid: "5093edb0975c4ca382fd6c7bac31237f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5093edb0975c4ca382fd6c7bac31237f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46085 } } }
I20260924 10:06:36.399518 2156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:36.403155 2163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:36.408397 2163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:36.417322 2163 catalog_manager.cc:1347] Generated new cluster ID: 88ce034134f64f3ea361b6c9d3cdb7bf
I20260924 10:06:36.417451 2163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:36.437958 2163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:36.438396 2163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:36.444274 2163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5093edb0975c4ca382fd6c7bac31237f: Generated new TSK 0
I20260924 10:06:36.445091 2163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:36.452719 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2077
W20260924 10:06:36.453802 2139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37093
W20260924 10:06:36.454550 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2077: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:36.454708 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2077
2026-09-24T10:06:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1778 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:36 dist-test-slave-jvsj krb5kdc[2175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:36 dist-test-slave-jvsj krb5kdc[2175](info): set up 2 sockets
Sep 24 10:06:36 dist-test-slave-jvsj krb5kdc[2175](info): commencing operation
krb5kdc: starting...
W20260924 10:06:37.470557 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:37 dist-test-slave-jvsj krb5kdc[2175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:37.705890 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:38.031406 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:38.031697 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:38.031795 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:38.039039 2188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:38.039160 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:38.039212 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:38.039271 2188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:38.039314 2188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:38.049180 2188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:38.050503 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:38.051819 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:38.063091 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:38.063215 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:38.063263 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:38.064425 2188 server_base.cc:1034] running on GCE node
I20260924 10:06:38.065369 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:38.066846 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:38.068112 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790244398068057 us; error 45 us; skew 500 ppm
I20260924 10:06:38.068455 2188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:38 dist-test-slave-jvsj krb5kdc[2175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:38.071646 2188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:38.073225 2188 webserver.cc:460] Webserver started at http://127.0.0.1:43045/ using document root <none> and password file <none>
I20260924 10:06:38.073910 2188 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:38.074034 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:38.074352 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:38.077425 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "b115f9de3fbf478bbfd1bd324bd31479"
format_stamp: "Formatted at 2026-09-24 10:06:38 on dist-test-slave-jvsj"
server_key: "700011fb1b0cafdeb96c336bfea5cdeb"
server_key_iv: "495fe697d4c672941a14bd7a44d74844"
server_key_version: "encryptionkey@0"
I20260924 10:06:38.078207 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "b115f9de3fbf478bbfd1bd324bd31479"
format_stamp: "Formatted at 2026-09-24 10:06:38 on dist-test-slave-jvsj"
server_key: "700011fb1b0cafdeb96c336bfea5cdeb"
server_key_iv: "495fe697d4c672941a14bd7a44d74844"
server_key_version: "encryptionkey@0"
I20260924 10:06:38.083433 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:06:38.087184 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:38.088611 2188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:38.088783 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "b115f9de3fbf478bbfd1bd324bd31479"
format_stamp: "Formatted at 2026-09-24 10:06:38 on dist-test-slave-jvsj"
server_key: "700011fb1b0cafdeb96c336bfea5cdeb"
server_key_iv: "495fe697d4c672941a14bd7a44d74844"
server_key_version: "encryptionkey@0"
I20260924 10:06:38.089018 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:38.112287 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:38.116556 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:38.116807 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:38.134357 2188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37343
I20260924 10:06:38.134392 2254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37343 every 8 connection(s)
I20260924 10:06:38.135731 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:38.138330 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2188
I20260924 10:06:38.138505 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:38.138918 13960 external_mini_cluster.cc:1428] Setting key 5a2a3bd1312685f493461941d48fe7c1
I20260924 10:06:38.139577 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:06:38 dist-test-slave-jvsj krb5kdc[2175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:06:38.151889 2255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479: Bootstrap starting.
I20260924 10:06:38.155655 2255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:38.156797 2255 log.cc:824] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:38.160147 2255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479: No bootstrap required, opened a new log
I20260924 10:06:38.160742 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:38.140639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59147 (local address 127.0.0.1:37343)
0924 10:06:38.141368 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:38.141380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:38.141420 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:06:38.143042 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:38.143067 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:38.143484 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:38.143880 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:38.145323 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:38.146712 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:38.147940 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:38.148616 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:38.148790 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:38.153124 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:38.153132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:38.153147 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:38.153194 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:38.156450 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:38.157320 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:38.157329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:38.157337 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:38.157590 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:38.157958 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:38.157962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:38.157966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:38.158576 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:38.158785 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:38.159338 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 10:06:38.159691 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":322,"threads_started":1}
I20260924 10:06:38.164354 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:38.169607 2255 raft_consensus.cc:348] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b115f9de3fbf478bbfd1bd324bd31479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37343 } }
I20260924 10:06:38.170030 2255 raft_consensus.cc:374] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:38.170135 2255 raft_consensus.cc:729] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b115f9de3fbf478bbfd1bd324bd31479, State: Initialized, Role: FOLLOWER
I20260924 10:06:38.170779 2255 consensus_queue.cc:260] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b115f9de3fbf478bbfd1bd324bd31479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37343 } }
I20260924 10:06:38.171001 2255 raft_consensus.cc:388] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:38.171135 2255 raft_consensus.cc:482] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:38.171293 2255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:38.173563 2255 raft_consensus.cc:504] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b115f9de3fbf478bbfd1bd324bd31479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37343 } }
I20260924 10:06:38.174096 2255 leader_election.cc:302] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b115f9de3fbf478bbfd1bd324bd31479; no voters:
I20260924 10:06:38.174741 2255 leader_election.cc:288] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:38.174832 2265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:38.179286 2255 sys_catalog.cc:564] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:38.179339 2265 raft_consensus.cc:686] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [term 1 LEADER]: Becoming Leader. State: Replica: b115f9de3fbf478bbfd1bd324bd31479, State: Running, Role: LEADER
I20260924 10:06:38.180145 2265 consensus_queue.cc:237] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b115f9de3fbf478bbfd1bd324bd31479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37343 } }
I20260924 10:06:38.185177 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b115f9de3fbf478bbfd1bd324bd31479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b115f9de3fbf478bbfd1bd324bd31479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37343 } } }
I20260924 10:06:38.185585 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:38.185664 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:38.169446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44055 (local address 127.0.0.1:37343)
0924 10:06:38.169629 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:38.169634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:38.169912 (+ 278us) server_negotiation.cc:407] Connection header received
0924 10:06:38.170119 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:38.170128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:38.170199 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:38.170287 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:38.171840 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:38.172611 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:38.173799 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:38.174257 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:38.174400 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:38.177639 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:38.177650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:38.177655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:38.177699 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:38.181270 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:38.182286 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:38.182292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:38.182296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:38.182486 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:38.182955 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:38.182963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:38.182967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:38.183395 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:38.183688 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:38.184938 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0924 10:06:38.185211 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:06:38.187422 2280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:38.187582 2267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b115f9de3fbf478bbfd1bd324bd31479. Latest consensus state: current_term: 1 leader_uuid: "b115f9de3fbf478bbfd1bd324bd31479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b115f9de3fbf478bbfd1bd324bd31479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37343 } } }
I20260924 10:06:38.187839 2267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:38.191895 2280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:38.200743 2280 catalog_manager.cc:1347] Generated new cluster ID: ef59a01e0d7a4cf5b949e4e554672ea5
I20260924 10:06:38.200860 2280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:38.225935 2280 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:38.227964 2280 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:38.244302 2280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b115f9de3fbf478bbfd1bd324bd31479: Generated new TSK 0
I20260924 10:06:38.245385 2280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:38.256677 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2188
I20260924 10:06:38.258563 2250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:38.351677 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2188
2026-09-24T10:06:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:38 dist-test-slave-jvsj krb5kdc[2287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:38 dist-test-slave-jvsj krb5kdc[2287](info): set up 2 sockets
Sep 24 10:06:38 dist-test-slave-jvsj krb5kdc[2287](info): commencing operation
krb5kdc: starting...
W20260924 10:06:39.374517 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:39 dist-test-slave-jvsj krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:39.611707 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:39.939365 2300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:39.939659 2300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:39.939776 2300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:39.947073 2300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:39.947187 2300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:39.947258 2300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:39.947332 2300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:39.947377 2300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:39.957239 2300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:39.958575 2300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:39.959838 2300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:39.971235 2308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:39.971421 2305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:39.971210 2306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:39.972205 2300 server_base.cc:1034] running on GCE node
I20260924 10:06:39.972934 2300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:39.974300 2300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:39.975541 2300 hybrid_clock.cc:648] HybridClock initialized: now 1790244399975489 us; error 39 us; skew 500 ppm
I20260924 10:06:39.975881 2300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:39 dist-test-slave-jvsj krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:39.979033 2300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:39.980409 2300 webserver.cc:460] Webserver started at http://127.0.0.1:35011/ using document root <none> and password file <none>
I20260924 10:06:39.981052 2300 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:39.981156 2300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:39.981406 2300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:39.984387 2300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "4fc4b71b63bb4d3bac5d2a47727d2402"
format_stamp: "Formatted at 2026-09-24 10:06:39 on dist-test-slave-jvsj"
server_key: "a872d5398f2fa1c4ef523e04575880d9"
server_key_iv: "f4c79df36d1fbbe2a1f5cac596a15272"
server_key_version: "encryptionkey@0"
I20260924 10:06:39.985100 2300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "4fc4b71b63bb4d3bac5d2a47727d2402"
format_stamp: "Formatted at 2026-09-24 10:06:39 on dist-test-slave-jvsj"
server_key: "a872d5398f2fa1c4ef523e04575880d9"
server_key_iv: "f4c79df36d1fbbe2a1f5cac596a15272"
server_key_version: "encryptionkey@0"
I20260924 10:06:39.990172 2300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 10:06:39.993625 2315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:39.995020 2300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:06:39.995164 2300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "4fc4b71b63bb4d3bac5d2a47727d2402"
format_stamp: "Formatted at 2026-09-24 10:06:39 on dist-test-slave-jvsj"
server_key: "a872d5398f2fa1c4ef523e04575880d9"
server_key_iv: "f4c79df36d1fbbe2a1f5cac596a15272"
server_key_version: "encryptionkey@0"
I20260924 10:06:39.995332 2300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:40.046468 2300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:40.050730 2300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:40.050974 2300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:40.067852 2300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38007
I20260924 10:06:40.067884 2366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38007 every 8 connection(s)
I20260924 10:06:40.069185 2300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:40.072814 2367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:40.075448 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2300
I20260924 10:06:40.075608 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:40.076006 13960 external_mini_cluster.cc:1428] Setting key 8258ff13a5058beec578142e7d72aaf3
I20260924 10:06:40.083590 2367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402: Bootstrap starting.
I20260924 10:06:40.086661 2367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:40.087802 2367 log.cc:824] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:40.089000 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:40.077803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59149 (local address 127.0.0.1:38007)
0924 10:06:40.078594 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:40.078606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:40.078646 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:06:40.080324 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:40.080347 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:40.080770 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:40.081149 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:40.081703 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:40.083065 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:40.084381 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:40.085105 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:40.085267 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:40.086395 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:40.086406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:40.086423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:40.087007 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:40.087134 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:40.087164 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:40.087272 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:40.087497 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 10:06:40.087755 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":338,"threads_started":1}
I20260924 10:06:40.090683 2367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402: No bootstrap required, opened a new log
I20260924 10:06:40.092530 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:40.098846 2367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260924 10:06:40.099282 2367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:40.099367 2367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc4b71b63bb4d3bac5d2a47727d2402, State: Initialized, Role: FOLLOWER
I20260924 10:06:40.099987 2367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260924 10:06:40.100212 2367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:40.100348 2367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:40.100533 2367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:40.102872 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:40.096667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58297 (local address 127.0.0.1:38007)
0924 10:06:40.096821 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:40.096827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:40.097319 (+ 492us) server_negotiation.cc:407] Connection header received
0924 10:06:40.097508 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:40.097512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:40.097600 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:40.097691 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:40.098162 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:40.098976 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:40.100381 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:40.101025 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:40.101170 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:40.102144 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:40.102150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:40.102155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:40.102179 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:40.102225 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:40.102245 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:40.102349 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:40.102475 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 10:06:40.102564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:06:40.103422 2367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260924 10:06:40.103930 2367 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc4b71b63bb4d3bac5d2a47727d2402; no voters:
I20260924 10:06:40.104537 2367 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:40.104643 2377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:40.105325 2377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc4b71b63bb4d3bac5d2a47727d2402, State: Running, Role: LEADER
I20260924 10:06:40.105872 2377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260924 10:06:40.106917 2367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:40.109076 2378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } } }
I20260924 10:06:40.109479 2378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:40.114238 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc4b71b63bb4d3bac5d2a47727d2402. Latest consensus state: current_term: 1 leader_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc4b71b63bb4d3bac5d2a47727d2402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } } }
I20260924 10:06:40.114615 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:40.115037 2385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:40.120232 2385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:40.129518 2385 catalog_manager.cc:1347] Generated new cluster ID: 14736db90c474c33a1fff8bc7d7d847a
I20260924 10:06:40.129642 2385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:40.161268 2385 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:40.163201 2385 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:40.187788 2385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc4b71b63bb4d3bac5d2a47727d2402: Generated new TSK 0
I20260924 10:06:40.188920 2385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:40.201238 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2300
W20260924 10:06:40.202505 2362 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59149
W20260924 10:06:40.203315 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:40.203475 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2300
2026-09-24T10:06:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:40 dist-test-slave-jvsj krb5kdc[2398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:40 dist-test-slave-jvsj krb5kdc[2398](info): set up 2 sockets
Sep 24 10:06:40 dist-test-slave-jvsj krb5kdc[2398](info): commencing operation
krb5kdc: starting...
W20260924 10:06:41.214459 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:41 dist-test-slave-jvsj krb5kdc[2398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:41.455925 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:41.787878 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:41.788204 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:41.788343 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:41.795686 2411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:41.795830 2411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:41.795909 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:41.795990 2411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:41.796056 2411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:41.806126 2411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:41.807493 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:41.808807 2411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:41.820298 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:41.820546 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:41.820287 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:41.821341 2411 server_base.cc:1034] running on GCE node
I20260924 10:06:41.822176 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:41.823741 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:41.824944 2411 hybrid_clock.cc:648] HybridClock initialized: now 1790244401824889 us; error 52 us; skew 500 ppm
I20260924 10:06:41.825282 2411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:41 dist-test-slave-jvsj krb5kdc[2398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:41.831666 2411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:41.833305 2411 webserver.cc:460] Webserver started at http://127.0.0.1:42217/ using document root <none> and password file <none>
I20260924 10:06:41.833966 2411 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:41.834074 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:41.834396 2411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:41.837466 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "b692a1ac234d4368b968ecf5aead4e21"
format_stamp: "Formatted at 2026-09-24 10:06:41 on dist-test-slave-jvsj"
server_key: "ccd260c6ce11f5f831b591de8bfadd8f"
server_key_iv: "03a3d189f16392f0993bfd9652fb6a78"
server_key_version: "encryptionkey@0"
I20260924 10:06:41.838212 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "b692a1ac234d4368b968ecf5aead4e21"
format_stamp: "Formatted at 2026-09-24 10:06:41 on dist-test-slave-jvsj"
server_key: "ccd260c6ce11f5f831b591de8bfadd8f"
server_key_iv: "03a3d189f16392f0993bfd9652fb6a78"
server_key_version: "encryptionkey@0"
I20260924 10:06:41.843621 2411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:06:41.847155 2426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:41.848562 2411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:41.848730 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "b692a1ac234d4368b968ecf5aead4e21"
format_stamp: "Formatted at 2026-09-24 10:06:41 on dist-test-slave-jvsj"
server_key: "ccd260c6ce11f5f831b591de8bfadd8f"
server_key_iv: "03a3d189f16392f0993bfd9652fb6a78"
server_key_version: "encryptionkey@0"
I20260924 10:06:41.848912 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:41.854640 2411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:41.858570 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:41.858855 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:41.876080 2411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46727
I20260924 10:06:41.876132 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46727 every 8 connection(s)
I20260924 10:06:41.877519 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:41.879367 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2411
I20260924 10:06:41.879513 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:41.879928 13960 external_mini_cluster.cc:1428] Setting key e6f84aece43bdfd21b9fbbf4a1d0f7a5
I20260924 10:06:41.881599 2478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:41.888515 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:41.881694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33781 (local address 127.0.0.1:46727)
0924 10:06:41.882505 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:41.882521 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:41.882575 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:06:41.884605 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:41.884633 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:41.885104 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:41.885541 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:41.886534 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:41.886542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:41.886558 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:41.886574 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:41.886664 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:41.886690 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:41.886910 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:41.887059 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:06:41.887251 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":311,"threads_started":1}
I20260924 10:06:41.892191 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:41.895728 2478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21: Bootstrap starting.
I20260924 10:06:41.899581 2478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:41.899771 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:41.897413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35259 (local address 127.0.0.1:46727)
0924 10:06:41.897611 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:41.897617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:41.897921 (+ 304us) server_negotiation.cc:407] Connection header received
0924 10:06:41.898072 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:41.898077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:41.898138 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:41.898245 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:41.899074 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:41.899079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:41.899084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:41.899102 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:41.899151 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:41.899171 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:41.899294 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:41.899363 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 10:06:41.899419 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:06:41.900868 2478 log.cc:824] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:41.903846 2478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21: No bootstrap required, opened a new log
I20260924 10:06:41.911370 2478 raft_consensus.cc:348] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692a1ac234d4368b968ecf5aead4e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260924 10:06:41.911753 2478 raft_consensus.cc:374] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:41.911837 2478 raft_consensus.cc:729] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b692a1ac234d4368b968ecf5aead4e21, State: Initialized, Role: FOLLOWER
I20260924 10:06:41.912376 2478 consensus_queue.cc:260] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692a1ac234d4368b968ecf5aead4e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260924 10:06:41.912571 2478 raft_consensus.cc:388] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:41.912669 2478 raft_consensus.cc:482] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:41.912796 2478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:41.915181 2478 raft_consensus.cc:504] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692a1ac234d4368b968ecf5aead4e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260924 10:06:41.915642 2478 leader_election.cc:302] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b692a1ac234d4368b968ecf5aead4e21; no voters:
I20260924 10:06:41.916131 2478 leader_election.cc:288] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:41.916262 2488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:41.916860 2488 raft_consensus.cc:686] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [term 1 LEADER]: Becoming Leader. State: Replica: b692a1ac234d4368b968ecf5aead4e21, State: Running, Role: LEADER
I20260924 10:06:41.917372 2488 consensus_queue.cc:237] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692a1ac234d4368b968ecf5aead4e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260924 10:06:41.918058 2478 sys_catalog.cc:564] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:41.920274 2489 sys_catalog.cc:455] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b692a1ac234d4368b968ecf5aead4e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692a1ac234d4368b968ecf5aead4e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } } }
I20260924 10:06:41.920670 2489 sys_catalog.cc:458] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:41.921182 2490 sys_catalog.cc:455] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b692a1ac234d4368b968ecf5aead4e21. Latest consensus state: current_term: 1 leader_uuid: "b692a1ac234d4368b968ecf5aead4e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692a1ac234d4368b968ecf5aead4e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } } }
I20260924 10:06:41.921466 2490 sys_catalog.cc:458] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:41.927079 2495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:41.932583 2495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:41.942175 2495 catalog_manager.cc:1347] Generated new cluster ID: 04fc54c431aa4909a6a6c17bca1b91f9
I20260924 10:06:41.942293 2495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:41.951879 2495 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:41.952373 2495 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:41.960018 2495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b692a1ac234d4368b968ecf5aead4e21: Generated new TSK 0
I20260924 10:06:41.961042 2495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:41.976651 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2411
W20260924 10:06:41.977826 2473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33781
W20260924 10:06:41.978581 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:41.978749 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2411
2026-09-24T10:06:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1774 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:42 dist-test-slave-jvsj krb5kdc[2509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:42 dist-test-slave-jvsj krb5kdc[2509](info): set up 2 sockets
Sep 24 10:06:42 dist-test-slave-jvsj krb5kdc[2509](info): commencing operation
krb5kdc: starting...
W20260924 10:06:42.990486 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:43 dist-test-slave-jvsj krb5kdc[2509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:43.233394 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:43.566447 2522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:43.566745 2522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:43.566885 2522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:43.574306 2522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:43.574460 2522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:43.574534 2522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:43.574627 2522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:43.574683 2522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:43.584897 2522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:43.586354 2522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:43.587687 2522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:43.599412 2527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:43.599339 2530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:43.599463 2528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:43.600533 2522 server_base.cc:1034] running on GCE node
I20260924 10:06:43.601292 2522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:43.602818 2522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:43.604031 2522 hybrid_clock.cc:648] HybridClock initialized: now 1790244403603984 us; error 40 us; skew 500 ppm
I20260924 10:06:43.604425 2522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:43 dist-test-slave-jvsj krb5kdc[2509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:43.607570 2522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:43.609179 2522 webserver.cc:460] Webserver started at http://127.0.0.1:46759/ using document root <none> and password file <none>
I20260924 10:06:43.609896 2522 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:43.610013 2522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:43.610304 2522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:43.613399 2522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "1e304c14dd894512817264d32c924f23"
format_stamp: "Formatted at 2026-09-24 10:06:43 on dist-test-slave-jvsj"
server_key: "23a39c40c93b061ea3eaba5fdb2eb960"
server_key_iv: "4597689ed4f81a5632eed28e5f404093"
server_key_version: "encryptionkey@0"
I20260924 10:06:43.614197 2522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "1e304c14dd894512817264d32c924f23"
format_stamp: "Formatted at 2026-09-24 10:06:43 on dist-test-slave-jvsj"
server_key: "23a39c40c93b061ea3eaba5fdb2eb960"
server_key_iv: "4597689ed4f81a5632eed28e5f404093"
server_key_version: "encryptionkey@0"
I20260924 10:06:43.619642 2522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:06:43.623296 2537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:43.624773 2522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:43.624946 2522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "1e304c14dd894512817264d32c924f23"
format_stamp: "Formatted at 2026-09-24 10:06:43 on dist-test-slave-jvsj"
server_key: "23a39c40c93b061ea3eaba5fdb2eb960"
server_key_iv: "4597689ed4f81a5632eed28e5f404093"
server_key_version: "encryptionkey@0"
I20260924 10:06:43.625146 2522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:43.647147 2522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:43.651583 2522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:43.651871 2522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:43.669633 2522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34409
I20260924 10:06:43.669664 2588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34409 every 8 connection(s)
I20260924 10:06:43.671046 2522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:43.674675 2589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:43.677623 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2522
I20260924 10:06:43.677807 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:43.678345 13960 external_mini_cluster.cc:1428] Setting key 0989b66ae3112c3489c09075f104934a
I20260924 10:06:43.686044 2589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23: Bootstrap starting.
I20260924 10:06:43.689023 2589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:43.690003 2589 log.cc:824] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:06:43 dist-test-slave-jvsj krb5kdc[2509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:06:43.692831 2589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23: No bootstrap required, opened a new log
I20260924 10:06:43.700340 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:43.680207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:48679 (local address 127.0.0.1:34409)
0924 10:06:43.680968 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:43.680980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:43.681021 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:06:43.682899 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:43.682923 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:43.683352 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:43.683754 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:43.685220 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:43.686605 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:43.687820 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:43.688477 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:43.688645 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:43.692871 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:43.692879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:43.692893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:43.692943 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:43.696105 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:43.696935 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:43.696945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:43.696952 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:43.697140 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:43.697578 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:43.697582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:43.697584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:43.698119 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:43.698464 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:43.698963 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 10:06:43.699270 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":327,"threads_started":1}
I20260924 10:06:43.701026 2589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e304c14dd894512817264d32c924f23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34409 } }
I20260924 10:06:43.701467 2589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:43.701570 2589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e304c14dd894512817264d32c924f23, State: Initialized, Role: FOLLOWER
I20260924 10:06:43.702157 2589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e304c14dd894512817264d32c924f23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34409 } }
I20260924 10:06:43.702414 2589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:43.702540 2589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:43.702701 2589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:43.704262 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:43.705344 2589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e304c14dd894512817264d32c924f23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34409 } }
I20260924 10:06:43.705901 2589 leader_election.cc:302] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e304c14dd894512817264d32c924f23; no voters:
I20260924 10:06:43.706549 2589 leader_election.cc:288] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:43.706631 2594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:43.708714 2589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:43.715220 2594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [term 1 LEADER]: Becoming Leader. State: Replica: 1e304c14dd894512817264d32c924f23, State: Running, Role: LEADER
I20260924 10:06:43.715930 2594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e304c14dd894512817264d32c924f23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34409 } }
I20260924 10:06:43.718797 2596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e304c14dd894512817264d32c924f23. Latest consensus state: current_term: 1 leader_uuid: "1e304c14dd894512817264d32c924f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e304c14dd894512817264d32c924f23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34409 } } }
I20260924 10:06:43.719226 2596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:43.718891 2595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e304c14dd894512817264d32c924f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e304c14dd894512817264d32c924f23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34409 } } }
I20260924 10:06:43.719726 2595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:43.724874 2608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:43.730772 2608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:43.738135 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:43.720908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:50861 (local address 127.0.0.1:34409)
0924 10:06:43.721087 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:43.721094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:43.722224 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 10:06:43.722384 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:43.722390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:43.722477 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:43.722601 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:43.724618 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:43.725986 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:43.727698 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:43.728327 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:43.728459 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:43.731939 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:43.731946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:43.731951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:43.731993 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:43.735286 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:43.736051 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:43.736057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:43.736061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:43.736178 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:43.736621 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:43.736627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:43.736630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:43.736967 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:43.737258 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:43.737622 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 10:06:43.737800 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:06:43.742396 2608 catalog_manager.cc:1347] Generated new cluster ID: 73165aeec5c544d1b1897319f29ad332
I20260924 10:06:43.742504 2608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:43.753129 2608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:43.754446 2608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:43.762526 2608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e304c14dd894512817264d32c924f23: Generated new TSK 0
I20260924 10:06:43.763356 2608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:43.768342 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2522
I20260924 10:06:43.769488 2584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:43.857354 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2522
W20260924 10:06:43.870983 2507 connection.cc:535] client connection to 127.0.0.1:34409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34409): Connection reset by peer (error 104)
2026-09-24T10:06:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:43 dist-test-slave-jvsj krb5kdc[2621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:43 dist-test-slave-jvsj krb5kdc[2621](info): set up 2 sockets
Sep 24 10:06:43 dist-test-slave-jvsj krb5kdc[2621](info): commencing operation
krb5kdc: starting...
W20260924 10:06:44.874609 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:45 dist-test-slave-jvsj krb5kdc[2621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:45.120886 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:45.447559 2634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:45.447850 2634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:45.447975 2634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:45.455319 2634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:45.455436 2634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:45.455652 2634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:45.455745 2634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:45.455809 2634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:45.466178 2634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:45.467825 2634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:45.469141 2634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:45.480549 2639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:45.480605 2640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:45.480734 2642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:45.481536 2634 server_base.cc:1034] running on GCE node
I20260924 10:06:45.482380 2634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:45.483788 2634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:45.484987 2634 hybrid_clock.cc:648] HybridClock initialized: now 1790244405484932 us; error 42 us; skew 500 ppm
I20260924 10:06:45.485313 2634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:45 dist-test-slave-jvsj krb5kdc[2621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:45.488379 2634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:45.489944 2634 webserver.cc:460] Webserver started at http://127.0.0.1:40161/ using document root <none> and password file <none>
I20260924 10:06:45.490661 2634 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:45.490780 2634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:45.491050 2634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:45.493994 2634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "4ef6f355fb0e48d9a9229f0d08fd8303"
format_stamp: "Formatted at 2026-09-24 10:06:45 on dist-test-slave-jvsj"
server_key: "5165cf1398046dc473f08343bc7e6dd5"
server_key_iv: "cbdc206f1f77073946930f9e37faea94"
server_key_version: "encryptionkey@0"
I20260924 10:06:45.494778 2634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "4ef6f355fb0e48d9a9229f0d08fd8303"
format_stamp: "Formatted at 2026-09-24 10:06:45 on dist-test-slave-jvsj"
server_key: "5165cf1398046dc473f08343bc7e6dd5"
server_key_iv: "cbdc206f1f77073946930f9e37faea94"
server_key_version: "encryptionkey@0"
I20260924 10:06:45.500295 2634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:06:45.504004 2649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:45.505488 2634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 10:06:45.505677 2634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "4ef6f355fb0e48d9a9229f0d08fd8303"
format_stamp: "Formatted at 2026-09-24 10:06:45 on dist-test-slave-jvsj"
server_key: "5165cf1398046dc473f08343bc7e6dd5"
server_key_iv: "cbdc206f1f77073946930f9e37faea94"
server_key_version: "encryptionkey@0"
I20260924 10:06:45.505872 2634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:45.520030 2634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:45.524307 2634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:45.524576 2634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:45.542286 2634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41499
I20260924 10:06:45.542344 2700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41499 every 8 connection(s)
I20260924 10:06:45.543740 2634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:45.547467 2701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:45.553961 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2634
I20260924 10:06:45.554136 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:45.554615 13960 external_mini_cluster.cc:1428] Setting key 7b4fe539b22e47ee59daa969965447ff
I20260924 10:06:45.558002 2701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303: Bootstrap starting.
I20260924 10:06:45.561292 2701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:45.562384 2701 log.cc:824] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:45.565256 2701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303: No bootstrap required, opened a new log
I20260924 10:06:45.568186 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:45.556399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:52199 (local address 127.0.0.1:41499)
0924 10:06:45.557284 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:45.557296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:45.557336 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:06:45.559274 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:45.559298 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:45.559745 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:45.560161 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:45.560673 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:45.562037 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:45.563703 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:45.564365 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:45.564497 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:45.565793 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:45.565800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:45.565813 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:45.566396 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:45.566474 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:45.566498 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:45.566616 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:45.566962 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 10:06:45.567167 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":434,"threads_started":1}
I20260924 10:06:45.571858 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:45.573797 2701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260924 10:06:45.574255 2701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:45.574385 2701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef6f355fb0e48d9a9229f0d08fd8303, State: Initialized, Role: FOLLOWER
I20260924 10:06:45.575068 2701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260924 10:06:45.575397 2701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:45.575544 2701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:45.575737 2701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:45.579257 2701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260924 10:06:45.579856 2701 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef6f355fb0e48d9a9229f0d08fd8303; no voters:
I20260924 10:06:45.580489 2701 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:45.580578 2711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:45.581710 2711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef6f355fb0e48d9a9229f0d08fd8303, State: Running, Role: LEADER
I20260924 10:06:45.582280 2711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260924 10:06:45.582932 2701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:45.584964 2712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } } }
I20260924 10:06:45.585346 2712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:45.587090 2713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef6f355fb0e48d9a9229f0d08fd8303. Latest consensus state: current_term: 1 leader_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef6f355fb0e48d9a9229f0d08fd8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } } }
I20260924 10:06:45.587502 2713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:45.588855 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:45.580009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:39999 (local address 127.0.0.1:41499)
0924 10:06:45.580212 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:45.580218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:45.580586 (+ 368us) server_negotiation.cc:407] Connection header received
0924 10:06:45.580882 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:45.580888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:45.580984 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:45.581106 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:45.581918 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:45.583110 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:45.584930 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:45.585628 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:45.585754 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:45.587460 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:45.587467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:45.587473 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:45.587503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:45.587581 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:45.587605 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:45.587706 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:45.588424 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 10:06:45.588490 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:06:45.590481 2716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:45.596221 2716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:45.606411 2716 catalog_manager.cc:1347] Generated new cluster ID: 0a548b53561e4789baa03f53035eb1f8
I20260924 10:06:45.606535 2716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:45.650347 2716 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:45.651665 2716 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:45.657976 2716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef6f355fb0e48d9a9229f0d08fd8303: Generated new TSK 0
I20260924 10:06:45.658815 2716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:45.673295 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2634
W20260924 10:06:45.674660 2696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.127:52199
W20260924 10:06:45.675400 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2634: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:45.675540 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2634
W20260924 10:06:45.689193 2619 connection.cc:535] client connection to 127.0.0.1:41499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41499): Connection reset by peer (error 104)
2026-09-24T10:06:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:45 dist-test-slave-jvsj krb5kdc[2732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:45 dist-test-slave-jvsj krb5kdc[2732](info): set up 2 sockets
Sep 24 10:06:45 dist-test-slave-jvsj krb5kdc[2732](info): commencing operation
krb5kdc: starting...
W20260924 10:06:46.682555 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:46 dist-test-slave-jvsj krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:46.919831 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:47.240186 2745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:47.240470 2745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:47.240588 2745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:47.248050 2745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:47.248173 2745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:47.248219 2745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:47.248270 2745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:47.248312 2745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:47.261057 2745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:47.262399 2745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:47.263716 2745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:47.274782 2753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:47.274856 2751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:47.274827 2750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:47.275650 2745 server_base.cc:1034] running on GCE node
I20260924 10:06:47.276399 2745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:47.277905 2745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:47.279181 2745 hybrid_clock.cc:648] HybridClock initialized: now 1790244407279080 us; error 44 us; skew 500 ppm
I20260924 10:06:47.279541 2745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:47 dist-test-slave-jvsj krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:47.288087 2745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:47.289701 2745 webserver.cc:460] Webserver started at http://127.0.0.1:34077/ using document root <none> and password file <none>
I20260924 10:06:47.290462 2745 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:47.290583 2745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:47.290885 2745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:47.294008 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "588bc708ee0d44de907c2f440c0a974d"
format_stamp: "Formatted at 2026-09-24 10:06:47 on dist-test-slave-jvsj"
server_key: "41f9537f9bafdb56e694fc149e0018d2"
server_key_iv: "85c2e2deb5e6b2c49e85154168adecc4"
server_key_version: "encryptionkey@0"
I20260924 10:06:47.294834 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "588bc708ee0d44de907c2f440c0a974d"
format_stamp: "Formatted at 2026-09-24 10:06:47 on dist-test-slave-jvsj"
server_key: "41f9537f9bafdb56e694fc149e0018d2"
server_key_iv: "85c2e2deb5e6b2c49e85154168adecc4"
server_key_version: "encryptionkey@0"
I20260924 10:06:47.300267 2745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:47.304018 2760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:47.305645 2745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:06:47.305819 2745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "588bc708ee0d44de907c2f440c0a974d"
format_stamp: "Formatted at 2026-09-24 10:06:47 on dist-test-slave-jvsj"
server_key: "41f9537f9bafdb56e694fc149e0018d2"
server_key_iv: "85c2e2deb5e6b2c49e85154168adecc4"
server_key_version: "encryptionkey@0"
I20260924 10:06:47.306038 2745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:47.312000 2745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:47.316138 2745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:47.316421 2745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:47.333853 2745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46149
I20260924 10:06:47.333889 2811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46149 every 8 connection(s)
I20260924 10:06:47.335713 2745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:47.339594 2812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:47.342839 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2745
I20260924 10:06:47.343014 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:47.343416 13960 external_mini_cluster.cc:1428] Setting key 6bd37955b185f17cccbed63eb42a32f8
I20260924 10:06:47.351151 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:47.345192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:57179 (local address 127.0.0.1:46149)
0924 10:06:47.345963 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:47.345976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:47.346018 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:06:47.347717 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:47.347742 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:47.348105 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:47.348500 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:47.349492 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:47.349500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:47.349513 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:47.349528 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:47.349598 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:47.349619 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:47.349828 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:47.349960 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 10:06:47.350162 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":309,"threads_started":1}
I20260924 10:06:47.351467 2812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d: Bootstrap starting.
I20260924 10:06:47.354758 2812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:47.354861 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:47.355794 2812 log.cc:824] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:47.358613 2812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d: No bootstrap required, opened a new log
I20260924 10:06:47.361523 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:47.359037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:50209 (local address 127.0.0.1:46149)
0924 10:06:47.359193 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:47.359199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:47.359667 (+ 468us) server_negotiation.cc:407] Connection header received
0924 10:06:47.359832 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:47.359838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:47.359899 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:47.359981 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:47.360870 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:47.360887 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:47.360891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:47.360907 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:47.360946 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:47.360960 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:47.361067 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:47.361164 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:06:47.361230 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:06:47.366560 2812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588bc708ee0d44de907c2f440c0a974d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260924 10:06:47.366930 2812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:47.367020 2812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bc708ee0d44de907c2f440c0a974d, State: Initialized, Role: FOLLOWER
I20260924 10:06:47.367547 2812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588bc708ee0d44de907c2f440c0a974d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260924 10:06:47.367844 2812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:47.367980 2812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:47.368158 2812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:47.370635 2812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588bc708ee0d44de907c2f440c0a974d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260924 10:06:47.371124 2812 leader_election.cc:302] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588bc708ee0d44de907c2f440c0a974d; no voters:
I20260924 10:06:47.371683 2812 leader_election.cc:288] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:47.371802 2822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:47.372517 2822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [term 1 LEADER]: Becoming Leader. State: Replica: 588bc708ee0d44de907c2f440c0a974d, State: Running, Role: LEADER
I20260924 10:06:47.373013 2822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588bc708ee0d44de907c2f440c0a974d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260924 10:06:47.373760 2812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:47.375612 2823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588bc708ee0d44de907c2f440c0a974d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588bc708ee0d44de907c2f440c0a974d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } } }
I20260924 10:06:47.376003 2823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:47.376634 2824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588bc708ee0d44de907c2f440c0a974d. Latest consensus state: current_term: 1 leader_uuid: "588bc708ee0d44de907c2f440c0a974d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588bc708ee0d44de907c2f440c0a974d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } } }
I20260924 10:06:47.376922 2824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:47.377859 2827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:47.383229 2827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:47.392191 2827 catalog_manager.cc:1347] Generated new cluster ID: 4a03ad9b34284b18907805e087120931
I20260924 10:06:47.392333 2827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:47.418079 2827 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:47.418530 2827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:47.429886 2827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588bc708ee0d44de907c2f440c0a974d: Generated new TSK 0
I20260924 10:06:47.431031 2827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:47.446233 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2745
W20260924 10:06:47.447475 2807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.127:57179
W20260924 10:06:47.448173 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:47.448308 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2745
2026-09-24T10:06:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1773 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:47 dist-test-slave-jvsj krb5kdc[2843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:47 dist-test-slave-jvsj krb5kdc[2843](info): set up 2 sockets
Sep 24 10:06:47 dist-test-slave-jvsj krb5kdc[2843](info): commencing operation
krb5kdc: starting...
W20260924 10:06:48.458613 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:48 dist-test-slave-jvsj krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:48.710168 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:49.037899 2856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:49.038237 2856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:49.038388 2856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:49.045717 2856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:49.045833 2856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:49.045879 2856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:49.045943 2856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:49.045984 2856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:49.055900 2856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:49.057168 2856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:49.058504 2856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:49.069852 2861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:49.069878 2862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:49.069900 2864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:49.070645 2856 server_base.cc:1034] running on GCE node
I20260924 10:06:49.071390 2856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:49.072781 2856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:49.074010 2856 hybrid_clock.cc:648] HybridClock initialized: now 1790244409073968 us; error 44 us; skew 500 ppm
I20260924 10:06:49.074409 2856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:49 dist-test-slave-jvsj krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:49.077414 2856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:49.078876 2856 webserver.cc:460] Webserver started at http://127.0.0.1:42267/ using document root <none> and password file <none>
I20260924 10:06:49.079535 2856 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:49.079640 2856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:49.079934 2856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:49.082906 2856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "419b97c71a9f4a4498545c2c7926bc66"
format_stamp: "Formatted at 2026-09-24 10:06:49 on dist-test-slave-jvsj"
server_key: "69bc657e0749f01d3b5cece4138d2c59"
server_key_iv: "3954fe1226ee0d2d9c5ad36afa20a094"
server_key_version: "encryptionkey@0"
I20260924 10:06:49.083654 2856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "419b97c71a9f4a4498545c2c7926bc66"
format_stamp: "Formatted at 2026-09-24 10:06:49 on dist-test-slave-jvsj"
server_key: "69bc657e0749f01d3b5cece4138d2c59"
server_key_iv: "3954fe1226ee0d2d9c5ad36afa20a094"
server_key_version: "encryptionkey@0"
I20260924 10:06:49.088863 2856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:06:49.092496 2871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:49.093892 2856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:06:49.094056 2856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "419b97c71a9f4a4498545c2c7926bc66"
format_stamp: "Formatted at 2026-09-24 10:06:49 on dist-test-slave-jvsj"
server_key: "69bc657e0749f01d3b5cece4138d2c59"
server_key_iv: "3954fe1226ee0d2d9c5ad36afa20a094"
server_key_version: "encryptionkey@0"
I20260924 10:06:49.094249 2856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:49.111891 2856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:49.116241 2856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:49.116515 2856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:49.133741 2856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40039
I20260924 10:06:49.133764 2922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40039 every 8 connection(s)
I20260924 10:06:49.135175 2856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:49.139101 2923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:49.143927 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2856
I20260924 10:06:49.144079 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:49.144503 13960 external_mini_cluster.cc:1428] Setting key 43964f542d63da371176c6ce39a70673
I20260924 10:06:49.150113 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66: Bootstrap starting.
I20260924 10:06:49.153282 2923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:49.154305 2923 log.cc:824] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:49.157044 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66: No bootstrap required, opened a new log
Sep 24 10:06:49 dist-test-slave-jvsj krb5kdc[2843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:06:49.165027 2923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419b97c71a9f4a4498545c2c7926bc66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260924 10:06:49.165534 2923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:49.165632 2923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419b97c71a9f4a4498545c2c7926bc66, State: Initialized, Role: FOLLOWER
I20260924 10:06:49.166217 2923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419b97c71a9f4a4498545c2c7926bc66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260924 10:06:49.166440 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:49.146340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:59595 (local address 127.0.0.1:40039)
0924 10:06:49.147100 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:49.147113 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:49.147157 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:06:49.148903 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:49.148927 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:49.149359 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:49.149719 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:49.151159 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:49.152497 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:49.153773 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:49.154544 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:49.154713 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:49.158804 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:49.158812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:49.158830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:49.158866 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:49.161961 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:49.162854 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:49.162861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:49.162868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:49.163036 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:49.163514 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:49.163518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:49.163520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:49.164108 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:49.164431 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:49.164933 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 10:06:49.165247 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":302,"threads_started":1}
I20260924 10:06:49.166517 2923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:49.166668 2923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:49.166857 2923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:49.169691 2923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419b97c71a9f4a4498545c2c7926bc66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260924 10:06:49.170238 2923 leader_election.cc:302] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419b97c71a9f4a4498545c2c7926bc66; no voters:
I20260924 10:06:49.170725 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:49.170900 2923 leader_election.cc:288] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:49.171011 2928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:49.173218 2923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:49.174875 2928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [term 1 LEADER]: Becoming Leader. State: Replica: 419b97c71a9f4a4498545c2c7926bc66, State: Running, Role: LEADER
I20260924 10:06:49.175612 2928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419b97c71a9f4a4498545c2c7926bc66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260924 10:06:49.178678 2929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "419b97c71a9f4a4498545c2c7926bc66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419b97c71a9f4a4498545c2c7926bc66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } } }
I20260924 10:06:49.179090 2929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:49.179124 2930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 419b97c71a9f4a4498545c2c7926bc66. Latest consensus state: current_term: 1 leader_uuid: "419b97c71a9f4a4498545c2c7926bc66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419b97c71a9f4a4498545c2c7926bc66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } } }
I20260924 10:06:49.179450 2930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:49.191571 2939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:49.197593 2939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:49.204829 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:49.182636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:43955 (local address 127.0.0.1:40039)
0924 10:06:49.182795 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:49.182802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:49.187137 (+ 4335us) server_negotiation.cc:407] Connection header received
0924 10:06:49.187338 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:49.187343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:49.187454 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:49.187601 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:49.189864 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:49.191407 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:49.192982 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:49.193738 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:49.193900 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:49.196837 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:49.196847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:49.196851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:49.196885 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:49.200758 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:49.201667 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:49.201673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:49.201677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:49.201810 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:49.202266 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:49.202271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:49.202274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:49.202755 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:49.203012 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:49.204287 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0924 10:06:49.204470 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:06:49.208487 2939 catalog_manager.cc:1347] Generated new cluster ID: 6f287df57f7b49129ae8c15fdd078b1e
I20260924 10:06:49.208613 2939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:49.247848 2939 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:49.249797 2939 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:49.262216 2939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 419b97c71a9f4a4498545c2c7926bc66: Generated new TSK 0
I20260924 10:06:49.263411 2939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:49.274785 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2856
I20260924 10:06:49.276015 2918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:49.371008 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2856
W20260924 10:06:49.385254 2841 connection.cc:535] client connection to 127.0.0.1:40039 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40039): Connection reset by peer (error 104)
2026-09-24T10:06:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1922 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:49 dist-test-slave-jvsj krb5kdc[2955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:49 dist-test-slave-jvsj krb5kdc[2955](info): set up 2 sockets
Sep 24 10:06:49 dist-test-slave-jvsj krb5kdc[2955](info): commencing operation
krb5kdc: starting...
W20260924 10:06:50.386587 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:50 dist-test-slave-jvsj krb5kdc[2955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:50.629391 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:50.957131 2968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:50.957435 2968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:50.957557 2968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:50.964646 2968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:50.964762 2968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:50.964818 2968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:50.964903 2968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:50.964959 2968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:50.974778 2968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:50.976089 2968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:50.977368 2968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:50.988307 2973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:50.988380 2976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:50.988364 2974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:50.989300 2968 server_base.cc:1034] running on GCE node
I20260924 10:06:50.990033 2968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:50.991427 2968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:50.992622 2968 hybrid_clock.cc:648] HybridClock initialized: now 1790244410992573 us; error 42 us; skew 500 ppm
I20260924 10:06:50.992957 2968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:50 dist-test-slave-jvsj krb5kdc[2955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:50.997382 2968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:51.000231 2968 webserver.cc:460] Webserver started at http://127.0.0.1:42213/ using document root <none> and password file <none>
I20260924 10:06:51.001117 2968 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:51.001243 2968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:51.001569 2968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:51.005611 2968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "e518cd14d9e1418b9d0bcba2beb2a17d"
format_stamp: "Formatted at 2026-09-24 10:06:50 on dist-test-slave-jvsj"
server_key: "d1a9543f0cf080e94df8e09baea941e0"
server_key_iv: "83c6748229b9dabfe6706d199cee9b44"
server_key_version: "encryptionkey@0"
I20260924 10:06:51.006661 2968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "e518cd14d9e1418b9d0bcba2beb2a17d"
format_stamp: "Formatted at 2026-09-24 10:06:50 on dist-test-slave-jvsj"
server_key: "d1a9543f0cf080e94df8e09baea941e0"
server_key_iv: "83c6748229b9dabfe6706d199cee9b44"
server_key_version: "encryptionkey@0"
I20260924 10:06:51.014285 2968 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 10:06:51.019160 2983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:51.020638 2968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20260924 10:06:51.020794 2968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "e518cd14d9e1418b9d0bcba2beb2a17d"
format_stamp: "Formatted at 2026-09-24 10:06:50 on dist-test-slave-jvsj"
server_key: "d1a9543f0cf080e94df8e09baea941e0"
server_key_iv: "83c6748229b9dabfe6706d199cee9b44"
server_key_version: "encryptionkey@0"
I20260924 10:06:51.020974 2968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:51.053525 2968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:51.057889 2968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:51.058125 2968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:51.074982 2968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41211
I20260924 10:06:51.075023 3034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41211 every 8 connection(s)
I20260924 10:06:51.076350 2968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:51.080096 3035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:51.083076 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 2968
I20260924 10:06:51.083258 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:51.083801 13960 external_mini_cluster.cc:1428] Setting key fb837e1526daaac367d2cab184836bca
I20260924 10:06:51.094864 3035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d: Bootstrap starting.
I20260924 10:06:51.099398 3035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:51.100868 3035 log.cc:824] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:51.102348 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:51.085798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:53193 (local address 127.0.0.1:41211)
0924 10:06:51.086687 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:51.086706 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:51.086760 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:06:51.089246 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:51.089284 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:51.089905 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:51.091209 (+ 1304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:51.091810 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:51.093908 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:51.096075 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:51.097311 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:51.098178 (+ 867us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:51.099408 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:51.099421 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:51.099440 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:51.100207 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:51.100309 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:51.100340 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:51.100583 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:51.100846 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 10:06:51.101050 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":382,"threads_started":1}
I20260924 10:06:51.104249 3035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d: No bootstrap required, opened a new log
I20260924 10:06:51.106474 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:51.112754 3035 raft_consensus.cc:348] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260924 10:06:51.113226 3035 raft_consensus.cc:374] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:51.113329 3035 raft_consensus.cc:729] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e518cd14d9e1418b9d0bcba2beb2a17d, State: Initialized, Role: FOLLOWER
I20260924 10:06:51.113902 3035 consensus_queue.cc:260] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260924 10:06:51.114115 3035 raft_consensus.cc:388] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:51.114239 3035 raft_consensus.cc:482] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:51.114419 3035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:51.116909 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:51.110641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:39009 (local address 127.0.0.1:41211)
0924 10:06:51.110817 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:51.110823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:51.111100 (+ 277us) server_negotiation.cc:407] Connection header received
0924 10:06:51.111251 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:51.111257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:51.111338 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:51.111453 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:51.111886 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:51.112952 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:51.114479 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:51.114941 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:51.115059 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:51.116184 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:51.116189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:51.116192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:51.116210 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:51.116249 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:51.116264 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:51.116357 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:51.116598 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:06:51.116651 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:06:51.116951 3035 raft_consensus.cc:504] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260924 10:06:51.117592 3035 leader_election.cc:302] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e518cd14d9e1418b9d0bcba2beb2a17d; no voters:
I20260924 10:06:51.118180 3035 leader_election.cc:288] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:51.118304 3045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:51.119304 3045 raft_consensus.cc:686] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [term 1 LEADER]: Becoming Leader. State: Replica: e518cd14d9e1418b9d0bcba2beb2a17d, State: Running, Role: LEADER
I20260924 10:06:51.119809 3045 consensus_queue.cc:237] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260924 10:06:51.120312 3035 sys_catalog.cc:564] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:51.122443 3047 sys_catalog.cc:455] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e518cd14d9e1418b9d0bcba2beb2a17d. Latest consensus state: current_term: 1 leader_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260924 10:06:51.122903 3047 sys_catalog.cc:458] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:51.123478 3046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e518cd14d9e1418b9d0bcba2beb2a17d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260924 10:06:51.123739 3046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:51.127676 3054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:51.131968 3054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:51.140215 3054 catalog_manager.cc:1347] Generated new cluster ID: 1da8696e613b4ebdba2b9e9471595932
I20260924 10:06:51.140328 3054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:51.157240 3054 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:51.159200 3054 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:51.171536 3054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e518cd14d9e1418b9d0bcba2beb2a17d: Generated new TSK 0
I20260924 10:06:51.172658 3054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:51.184559 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2968
W20260924 10:06:51.185947 3030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.127:53193
W20260924 10:06:51.186729 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 2968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:51.186900 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 2968
W20260924 10:06:51.201272 2953 connection.cc:535] client connection to 127.0.0.1:41211 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41211): Connection reset by peer (error 104)
2026-09-24T10:06:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1820 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:51 dist-test-slave-jvsj krb5kdc[3066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:51 dist-test-slave-jvsj krb5kdc[3066](info): set up 2 sockets
Sep 24 10:06:51 dist-test-slave-jvsj krb5kdc[3066](info): commencing operation
krb5kdc: starting...
W20260924 10:06:52.198585 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:52 dist-test-slave-jvsj krb5kdc[3066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:06:52.437345 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:52.769007 3079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:52.769300 3079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:52.769430 3079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:52.776855 3079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:52.777000 3079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:52.777058 3079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:52.777143 3079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:52.777204 3079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:52.787945 3079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:52.789355 3079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:52.790747 3079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:52.802611 3087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:52.802644 3085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:52.802583 3084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:52.803632 3079 server_base.cc:1034] running on GCE node
I20260924 10:06:52.804418 3079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:52.805940 3079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:52.807154 3079 hybrid_clock.cc:648] HybridClock initialized: now 1790244412807103 us; error 42 us; skew 500 ppm
I20260924 10:06:52.807509 3079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:52 dist-test-slave-jvsj krb5kdc[3066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:52.815174 3079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:06:52.816675 3079 webserver.cc:460] Webserver started at http://127.0.0.1:33889/ using document root <none> and password file <none>
I20260924 10:06:52.817399 3079 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:52.817528 3079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:52.817828 3079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:52.821031 3079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "a572654405924f21982dad543bb7e6da"
format_stamp: "Formatted at 2026-09-24 10:06:52 on dist-test-slave-jvsj"
server_key: "4d5f313a749b8b8cde9cd6efc252dbce"
server_key_iv: "9bc382f66cbf0a3d52c343b115f46104"
server_key_version: "encryptionkey@0"
I20260924 10:06:52.821813 3079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "a572654405924f21982dad543bb7e6da"
format_stamp: "Formatted at 2026-09-24 10:06:52 on dist-test-slave-jvsj"
server_key: "4d5f313a749b8b8cde9cd6efc252dbce"
server_key_iv: "9bc382f66cbf0a3d52c343b115f46104"
server_key_version: "encryptionkey@0"
I20260924 10:06:52.827236 3079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 10:06:52.830801 3094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:52.832202 3079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:06:52.832372 3079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "a572654405924f21982dad543bb7e6da"
format_stamp: "Formatted at 2026-09-24 10:06:52 on dist-test-slave-jvsj"
server_key: "4d5f313a749b8b8cde9cd6efc252dbce"
server_key_iv: "9bc382f66cbf0a3d52c343b115f46104"
server_key_version: "encryptionkey@0"
I20260924 10:06:52.832582 3079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:52.838295 3079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:52.842732 3079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:52.843024 3079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:52.861910 3079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43641
I20260924 10:06:52.861966 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43641 every 8 connection(s)
I20260924 10:06:52.863411 3079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:52.867427 3146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:52.870440 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3079
I20260924 10:06:52.870645 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:52.871224 13960 external_mini_cluster.cc:1428] Setting key 67751b105eb1a1a6f4b6fcc5e878f1e4
I20260924 10:06:52.879348 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:52.873253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:46739 (local address 127.0.0.1:43641)
0924 10:06:52.874080 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:52.874102 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:52.874152 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:06:52.875885 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:52.875911 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:52.876287 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:52.876676 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:52.877687 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:52.877694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:52.877711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:52.877732 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:52.877790 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:52.877819 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:52.878029 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:52.878157 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 10:06:52.878391 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":347,"threads_started":1}
I20260924 10:06:52.879834 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da: Bootstrap starting.
I20260924 10:06:52.883517 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:52.883682 3146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:52.884701 3146 log.cc:824] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:52.887621 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da: No bootstrap required, opened a new log
I20260924 10:06:52.890156 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:52.887718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:45719 (local address 127.0.0.1:43641)
0924 10:06:52.887909 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:52.887915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:52.888284 (+ 369us) server_negotiation.cc:407] Connection header received
0924 10:06:52.888458 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:52.888462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:52.888516 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:52.888597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:52.889528 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:52.889532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:52.889538 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:52.889554 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:52.889592 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:52.889609 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:52.889702 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:52.889822 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:06:52.889875 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:06:52.896471 3146 raft_consensus.cc:348] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572654405924f21982dad543bb7e6da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43641 } }
I20260924 10:06:52.896842 3146 raft_consensus.cc:374] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:52.896939 3146 raft_consensus.cc:729] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a572654405924f21982dad543bb7e6da, State: Initialized, Role: FOLLOWER
I20260924 10:06:52.897560 3146 consensus_queue.cc:260] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572654405924f21982dad543bb7e6da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43641 } }
I20260924 10:06:52.897783 3146 raft_consensus.cc:388] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:52.897918 3146 raft_consensus.cc:482] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:52.898073 3146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:52.900339 3146 raft_consensus.cc:504] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572654405924f21982dad543bb7e6da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43641 } }
I20260924 10:06:52.900833 3146 leader_election.cc:302] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a572654405924f21982dad543bb7e6da; no voters:
I20260924 10:06:52.901388 3146 leader_election.cc:288] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:52.901510 3156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:52.902535 3156 raft_consensus.cc:686] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [term 1 LEADER]: Becoming Leader. State: Replica: a572654405924f21982dad543bb7e6da, State: Running, Role: LEADER
I20260924 10:06:52.903163 3156 consensus_queue.cc:237] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572654405924f21982dad543bb7e6da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43641 } }
I20260924 10:06:52.903570 3146 sys_catalog.cc:564] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:52.905761 3158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [sys.catalog]: SysCatalogTable state changed. Reason: New leader a572654405924f21982dad543bb7e6da. Latest consensus state: current_term: 1 leader_uuid: "a572654405924f21982dad543bb7e6da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572654405924f21982dad543bb7e6da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43641 } } }
I20260924 10:06:52.906167 3158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:52.906730 3157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a572654405924f21982dad543bb7e6da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572654405924f21982dad543bb7e6da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43641 } } }
I20260924 10:06:52.907040 3157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:52.908320 3163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:52.912698 3163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:52.921452 3163 catalog_manager.cc:1347] Generated new cluster ID: c5e17ecb5e7d46228e1f31d44d4625b0
I20260924 10:06:52.921618 3163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:52.941214 3163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:52.941620 3163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:52.952915 3163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a572654405924f21982dad543bb7e6da: Generated new TSK 0
I20260924 10:06:52.953756 3163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:52.971396 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3079
W20260924 10:06:52.972503 3141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.127:46739
W20260924 10:06:52.973201 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3079: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:52.973459 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3079
2026-09-24T10:06:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1781 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:53 dist-test-slave-jvsj krb5kdc[3178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:53 dist-test-slave-jvsj krb5kdc[3178](info): set up 2 sockets
Sep 24 10:06:53 dist-test-slave-jvsj krb5kdc[3178](info): commencing operation
krb5kdc: starting...
W20260924 10:06:53.982547 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:54 dist-test-slave-jvsj krb5kdc[3178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:54.225428 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44129
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:54.553414 3191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:54.553730 3191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:54.553864 3191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:54.561244 3191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:54.561359 3191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:54.561430 3191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:54.561524 3191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:54.561579 3191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:54.571484 3191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:54.572806 3191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:54.574116 3191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:54.585515 3199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:54.585646 3196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:54.585515 3197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:54.586302 3191 server_base.cc:1034] running on GCE node
I20260924 10:06:54.587082 3191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:54.588546 3191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:54.589735 3191 hybrid_clock.cc:648] HybridClock initialized: now 1790244414589689 us; error 37 us; skew 500 ppm
I20260924 10:06:54.590135 3191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:54 dist-test-slave-jvsj krb5kdc[3178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244414, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:54.593209 3191 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:06:54.594689 3191 webserver.cc:460] Webserver started at http://127.13.162.62:40269/ using document root <none> and password file <none>
I20260924 10:06:54.595379 3191 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:54.595496 3191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:54.595779 3191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:54.598800 3191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "3594d477561444f7b531d7e66c61c6e5"
format_stamp: "Formatted at 2026-09-24 10:06:54 on dist-test-slave-jvsj"
server_key: "d5a3b3220dfdcf6ab7f1f4ae0a476bc4"
server_key_iv: "9b31aca7c34e59cf1b40213afb735faf"
server_key_version: "encryptionkey@0"
I20260924 10:06:54.599555 3191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "3594d477561444f7b531d7e66c61c6e5"
format_stamp: "Formatted at 2026-09-24 10:06:54 on dist-test-slave-jvsj"
server_key: "d5a3b3220dfdcf6ab7f1f4ae0a476bc4"
server_key_iv: "9b31aca7c34e59cf1b40213afb735faf"
server_key_version: "encryptionkey@0"
I20260924 10:06:54.604746 3191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:06:54.608368 3206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:54.609836 3191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:06:54.610035 3191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "3594d477561444f7b531d7e66c61c6e5"
format_stamp: "Formatted at 2026-09-24 10:06:54 on dist-test-slave-jvsj"
server_key: "d5a3b3220dfdcf6ab7f1f4ae0a476bc4"
server_key_iv: "9b31aca7c34e59cf1b40213afb735faf"
server_key_version: "encryptionkey@0"
I20260924 10:06:54.610224 3191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:54.622766 3191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:54.627084 3191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:54.627324 3191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:54.644214 3191 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:44129
I20260924 10:06:54.644224 3257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:44129 every 8 connection(s)
I20260924 10:06:54.645601 3191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:54.649444 3258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:54.649533 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3191
I20260924 10:06:54.649708 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:54.650254 13960 external_mini_cluster.cc:1428] Setting key ff89990827d7e5409ddbde84206d41ee
I20260924 10:06:54.663937 3258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5: Bootstrap starting.
Sep 24 10:06:54 dist-test-slave-jvsj krb5kdc[3178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:06:54.667722 3258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:54.669054 3258 log.cc:824] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:54.672498 3258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5: No bootstrap required, opened a new log
I20260924 10:06:54.673903 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:54.652060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51215 (local address 127.13.162.62:44129)
0924 10:06:54.652866 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:54.652880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:54.652926 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:06:54.654520 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:54.654542 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:54.654950 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:54.655290 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:54.657006 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:54.658282 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:54.660028 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:54.660733 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:54.660864 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:54.665567 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:54.665579 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:54.665599 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:54.665653 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:54.669301 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:54.670201 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:54.670207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:54.670215 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:54.670435 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:54.670865 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:54.670870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:54.670873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:54.671562 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:54.671864 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:54.672198 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 10:06:54.672557 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":374,"threads_started":1}
I20260924 10:06:54.677841 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:54.682101 3258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594d477561444f7b531d7e66c61c6e5" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44129 } }
I20260924 10:06:54.682577 3258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:54.682691 3258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3594d477561444f7b531d7e66c61c6e5, State: Initialized, Role: FOLLOWER
I20260924 10:06:54.683326 3258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594d477561444f7b531d7e66c61c6e5" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44129 } }
I20260924 10:06:54.683543 3258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:54.683672 3258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:54.683818 3258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:54.685945 3258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594d477561444f7b531d7e66c61c6e5" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44129 } }
I20260924 10:06:54.686483 3258 leader_election.cc:302] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3594d477561444f7b531d7e66c61c6e5; no voters:
I20260924 10:06:54.687085 3258 leader_election.cc:288] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:54.687208 3268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:54.688966 3268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [term 1 LEADER]: Becoming Leader. State: Replica: 3594d477561444f7b531d7e66c61c6e5, State: Running, Role: LEADER
I20260924 10:06:54.689551 3258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:54.689620 3268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594d477561444f7b531d7e66c61c6e5" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44129 } }
I20260924 10:06:54.692528 3270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3594d477561444f7b531d7e66c61c6e5. Latest consensus state: current_term: 1 leader_uuid: "3594d477561444f7b531d7e66c61c6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594d477561444f7b531d7e66c61c6e5" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44129 } } }
I20260924 10:06:54.692924 3270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:54.693465 3269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3594d477561444f7b531d7e66c61c6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594d477561444f7b531d7e66c61c6e5" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44129 } } }
I20260924 10:06:54.693768 3269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:54.702018 3274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:54.704229 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:54.682925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37603 (local address 127.13.162.62:44129)
0924 10:06:54.683096 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:54.683101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:54.686701 (+ 3600us) server_negotiation.cc:407] Connection header received
0924 10:06:54.686874 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:54.686880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:54.686949 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:54.687046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:54.688757 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:54.689751 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:54.691078 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:54.691681 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:54.691810 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:54.695185 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:54.695192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:54.695197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:06:54.695242 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:54.700154 (+ 4912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:54.701272 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:54.701277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:54.701281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:54.701436 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:06:54.702056 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:06:54.702062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:06:54.702064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:06:54.702565 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:54.702849 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:54.703394 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 10:06:54.703587 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:06:54.707255 3274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:54.715555 3274 catalog_manager.cc:1347] Generated new cluster ID: 0af1e74d19064b89ae4cd9d91c0920d6
I20260924 10:06:54.715664 3274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:54.738031 3274 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:54.740152 3274 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:54.750864 3274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3594d477561444f7b531d7e66c61c6e5: Generated new TSK 0
I20260924 10:06:54.751730 3274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:54.763154 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3191
I20260924 10:06:54.764926 3253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:06:54.856251 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3191
W20260924 10:06:54.870246 3176 connection.cc:535] client connection to 127.13.162.62:44129 recv error: Network error: failed to read from TLS socket (remote: 127.13.162.62:44129): Connection reset by peer (error 104)
2026-09-24T10:06:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:54 dist-test-slave-jvsj krb5kdc[3291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:54 dist-test-slave-jvsj krb5kdc[3291](info): set up 2 sockets
Sep 24 10:06:54 dist-test-slave-jvsj krb5kdc[3291](info): commencing operation
krb5kdc: starting...
W20260924 10:06:55.874562 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:56 dist-test-slave-jvsj krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:56.117689 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:41589
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:41589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:56.448184 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:56.448553 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:56.448704 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:56.456270 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:56.456404 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:56.456459 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:56.456522 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:56.456568 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:56.468593 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:41589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:56.470419 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:56.471700 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:56.483281 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:56.483206 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:56.483628 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:56.484093 3304 server_base.cc:1034] running on GCE node
I20260924 10:06:56.484917 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:56.486454 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:56.487677 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790244416487612 us; error 49 us; skew 500 ppm
I20260924 10:06:56.488044 3304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:56 dist-test-slave-jvsj krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244416, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:56.491235 3304 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:06:56.492772 3304 webserver.cc:460] Webserver started at http://127.13.162.62:44567/ using document root <none> and password file <none>
I20260924 10:06:56.493469 3304 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:56.493589 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:56.493881 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:56.497038 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "7da5b8f3183840dfbb840938c34d7fd8"
format_stamp: "Formatted at 2026-09-24 10:06:56 on dist-test-slave-jvsj"
server_key: "a3a2a4da2df5182f119dd6c72afd6582"
server_key_iv: "8f73fced51f69ec8d66e6028c90f8cfa"
server_key_version: "encryptionkey@0"
I20260924 10:06:56.497931 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "7da5b8f3183840dfbb840938c34d7fd8"
format_stamp: "Formatted at 2026-09-24 10:06:56 on dist-test-slave-jvsj"
server_key: "a3a2a4da2df5182f119dd6c72afd6582"
server_key_iv: "8f73fced51f69ec8d66e6028c90f8cfa"
server_key_version: "encryptionkey@0"
I20260924 10:06:56.503419 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:06:56.507254 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:56.508742 3304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:06:56.508922 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "7da5b8f3183840dfbb840938c34d7fd8"
format_stamp: "Formatted at 2026-09-24 10:06:56 on dist-test-slave-jvsj"
server_key: "a3a2a4da2df5182f119dd6c72afd6582"
server_key_iv: "8f73fced51f69ec8d66e6028c90f8cfa"
server_key_version: "encryptionkey@0"
I20260924 10:06:56.509122 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:56.524811 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:56.529143 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:56.529412 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:56.547617 3304 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:41589
I20260924 10:06:56.547643 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:41589 every 8 connection(s)
I20260924 10:06:56.549108 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:56.550787 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3304
I20260924 10:06:56.550974 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:56.551489 13960 external_mini_cluster.cc:1428] Setting key 89888ef007df32053bb7fced00d74fa8
I20260924 10:06:56.553959 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:56.565915 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:56.553393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55451 (local address 127.13.162.62:41589)
0924 10:06:56.554335 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:56.554353 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:56.554679 (+ 326us) server_negotiation.cc:407] Connection header received
0924 10:06:56.556481 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:56.556505 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:56.556950 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:56.557295 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:56.557916 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:56.559357 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:56.561190 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:56.561913 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:56.562044 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:56.563373 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:56.563381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:56.563400 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:56.564024 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:56.564111 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:56.564140 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:56.564289 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:56.564602 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 10:06:56.564828 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":365,"threads_started":1}
I20260924 10:06:56.567209 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8: Bootstrap starting.
I20260924 10:06:56.570077 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:56.570703 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:56.571727 3371 log.cc:824] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:56.575045 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8: No bootstrap required, opened a new log
I20260924 10:06:56.580760 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:56.574285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34909 (local address 127.13.162.62:41589)
0924 10:06:56.574454 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:56.574459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:56.574937 (+ 478us) server_negotiation.cc:407] Connection header received
0924 10:06:56.575117 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:56.575123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:56.575198 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:56.575300 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:56.575779 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:56.576905 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:56.578187 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:06:56.578891 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:06:56.579025 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:06:56.580042 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:56.580048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:56.580053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:56.580075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:56.580124 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:56.580147 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:56.580268 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:56.580365 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:06:56.580422 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:06:56.584146 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da5b8f3183840dfbb840938c34d7fd8" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41589 } }
I20260924 10:06:56.584583 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:56.584702 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da5b8f3183840dfbb840938c34d7fd8, State: Initialized, Role: FOLLOWER
I20260924 10:06:56.585228 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da5b8f3183840dfbb840938c34d7fd8" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41589 } }
I20260924 10:06:56.585444 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:56.585543 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:56.585672 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:56.588044 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da5b8f3183840dfbb840938c34d7fd8" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41589 } }
I20260924 10:06:56.588665 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7da5b8f3183840dfbb840938c34d7fd8; no voters:
I20260924 10:06:56.589327 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:56.589447 3381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:56.590158 3381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [term 1 LEADER]: Becoming Leader. State: Replica: 7da5b8f3183840dfbb840938c34d7fd8, State: Running, Role: LEADER
I20260924 10:06:56.590703 3381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da5b8f3183840dfbb840938c34d7fd8" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41589 } }
I20260924 10:06:56.591463 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:56.593274 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7da5b8f3183840dfbb840938c34d7fd8. Latest consensus state: current_term: 1 leader_uuid: "7da5b8f3183840dfbb840938c34d7fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da5b8f3183840dfbb840938c34d7fd8" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41589 } } }
I20260924 10:06:56.593684 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:56.593536 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7da5b8f3183840dfbb840938c34d7fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da5b8f3183840dfbb840938c34d7fd8" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 41589 } } }
I20260924 10:06:56.593827 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:56.595742 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:56.600869 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:56.610188 3389 catalog_manager.cc:1347] Generated new cluster ID: 9a1f96d926514ee8911d3df2a3f743d3
I20260924 10:06:56.610291 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:56.627803 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:56.629139 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:56.643770 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7da5b8f3183840dfbb840938c34d7fd8: Generated new TSK 0
I20260924 10:06:56.644650 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:56.658653 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3304
W20260924 10:06:56.659696 3366 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55451
W20260924 10:06:56.660398 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3304: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:56.660522 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3304
W20260924 10:06:56.674228 3289 connection.cc:535] client connection to 127.13.162.62:41589 recv error: Network error: failed to read from TLS socket (remote: 127.13.162.62:41589): Connection reset by peer (error 104)
2026-09-24T10:06:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1803 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:56 dist-test-slave-jvsj krb5kdc[3402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:56 dist-test-slave-jvsj krb5kdc[3402](info): set up 2 sockets
Sep 24 10:06:56 dist-test-slave-jvsj krb5kdc[3402](info): commencing operation
krb5kdc: starting...
W20260924 10:06:57.678678 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:57 dist-test-slave-jvsj krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:57.918835 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45711
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:45711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:06:58.240458 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:06:58.240758 3415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:06:58.240895 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:06:58.248214 3415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:06:58.248348 3415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:06:58.248428 3415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:06:58.248514 3415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:06:58.248577 3415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:06:58.258760 3415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:45711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:45711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:06:58.260092 3415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:06:58.261415 3415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:06:58.273065 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:58.273106 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:06:58.273027 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:06:58.274120 3415 server_base.cc:1034] running on GCE node
I20260924 10:06:58.274978 3415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:06:58.276441 3415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:06:58.277683 3415 hybrid_clock.cc:648] HybridClock initialized: now 1790244418277614 us; error 61 us; skew 500 ppm
I20260924 10:06:58.278065 3415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:06:58 dist-test-slave-jvsj krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244418, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:06:58.284627 3415 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:06:58.286291 3415 webserver.cc:460] Webserver started at http://127.13.162.62:43259/ using document root <none> and password file <none>
I20260924 10:06:58.287048 3415 fs_manager.cc:359] Metadata directory not provided
I20260924 10:06:58.287174 3415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:06:58.287458 3415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:06:58.290642 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "c4d61a3a650c44fc8110fba83ade48b1"
format_stamp: "Formatted at 2026-09-24 10:06:58 on dist-test-slave-jvsj"
server_key: "977cec244c1a404bd4bff3ee81222d60"
server_key_iv: "8b62c23fd86c1a83b1f1187020e25562"
server_key_version: "encryptionkey@0"
I20260924 10:06:58.291410 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "c4d61a3a650c44fc8110fba83ade48b1"
format_stamp: "Formatted at 2026-09-24 10:06:58 on dist-test-slave-jvsj"
server_key: "977cec244c1a404bd4bff3ee81222d60"
server_key_iv: "8b62c23fd86c1a83b1f1187020e25562"
server_key_version: "encryptionkey@0"
I20260924 10:06:58.296667 3415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 10:06:58.300299 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:06:58.301739 3415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:06:58.301915 3415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "c4d61a3a650c44fc8110fba83ade48b1"
format_stamp: "Formatted at 2026-09-24 10:06:58 on dist-test-slave-jvsj"
server_key: "977cec244c1a404bd4bff3ee81222d60"
server_key_iv: "8b62c23fd86c1a83b1f1187020e25562"
server_key_version: "encryptionkey@0"
I20260924 10:06:58.302122 3415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:06:58.308002 3415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:06:58.312270 3415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:06:58.312556 3415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:06:58.329636 3415 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:45711
I20260924 10:06:58.329654 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:45711 every 8 connection(s)
I20260924 10:06:58.331058 3415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:06:58.334663 3482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:06:58.340886 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3415
I20260924 10:06:58.341056 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:06:58.341456 13960 external_mini_cluster.cc:1428] Setting key bd56c60e66306a61fe95d9c4ab08074a
I20260924 10:06:58.345484 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1: Bootstrap starting.
I20260924 10:06:58.348735 3482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:06:58.349040 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:58.343255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45001 (local address 127.13.162.62:45711)
0924 10:06:58.344071 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:58.344083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:58.344128 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:06:58.345769 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:58.345794 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:58.346148 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:58.346532 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:58.347503 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:58.347509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:58.347523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:58.347536 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:58.347597 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:58.347620 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:58.347814 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:58.347945 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:06:58.348122 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":348,"threads_started":1}
I20260924 10:06:58.349995 3482 log.cc:824] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:06:58.352934 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:06:58.353096 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1: No bootstrap required, opened a new log
I20260924 10:06:58.359441 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:06:58.356998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52387 (local address 127.13.162.62:45711)
0924 10:06:58.357163 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:06:58.357169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:06:58.357569 (+ 400us) server_negotiation.cc:407] Connection header received
0924 10:06:58.357739 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:06:58.357743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:06:58.357791 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:06:58.357876 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:06:58.358828 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:06:58.358834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:06:58.358838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:06:58.358856 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:06:58.358894 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:06:58.358908 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:06:58.359012 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:06:58.359124 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 10:06:58.359184 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:06:58.361151 3482 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d61a3a650c44fc8110fba83ade48b1" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45711 } }
I20260924 10:06:58.361485 3482 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:06:58.361578 3482 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d61a3a650c44fc8110fba83ade48b1, State: Initialized, Role: FOLLOWER
I20260924 10:06:58.362106 3482 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d61a3a650c44fc8110fba83ade48b1" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45711 } }
I20260924 10:06:58.362344 3482 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:06:58.362473 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:06:58.362623 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:06:58.364710 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d61a3a650c44fc8110fba83ade48b1" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45711 } }
I20260924 10:06:58.365187 3482 leader_election.cc:302] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d61a3a650c44fc8110fba83ade48b1; no voters:
I20260924 10:06:58.365828 3482 leader_election.cc:288] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:06:58.365931 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:06:58.366583 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [term 1 LEADER]: Becoming Leader. State: Replica: c4d61a3a650c44fc8110fba83ade48b1, State: Running, Role: LEADER
I20260924 10:06:58.367128 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d61a3a650c44fc8110fba83ade48b1" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45711 } }
I20260924 10:06:58.368126 3482 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:06:58.370352 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d61a3a650c44fc8110fba83ade48b1. Latest consensus state: current_term: 1 leader_uuid: "c4d61a3a650c44fc8110fba83ade48b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d61a3a650c44fc8110fba83ade48b1" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45711 } } }
I20260924 10:06:58.370764 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:58.376010 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d61a3a650c44fc8110fba83ade48b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d61a3a650c44fc8110fba83ade48b1" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 45711 } } }
I20260924 10:06:58.376302 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:06:58.378985 3500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:06:58.384207 3500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:06:58.393597 3500 catalog_manager.cc:1347] Generated new cluster ID: 4b77c05d843b439c88b412105f362a8e
I20260924 10:06:58.393704 3500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:06:58.421693 3500 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:06:58.422204 3500 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:06:58.438403 3500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d61a3a650c44fc8110fba83ade48b1: Generated new TSK 0
I20260924 10:06:58.439471 3500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:06:58.451463 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3415
W20260924 10:06:58.452615 3477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45001
W20260924 10:06:58.453384 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3415: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:06:58.453549 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3415
2026-09-24T10:06:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1794 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:06:58 dist-test-slave-jvsj krb5kdc[3513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:06:58 dist-test-slave-jvsj krb5kdc[3513](info): set up 2 sockets
Sep 24 10:06:58 dist-test-slave-jvsj krb5kdc[3513](info): commencing operation
krb5kdc: starting...
W20260924 10:06:59.468290 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:06:59 dist-test-slave-jvsj krb5kdc[3513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:06:59.707561 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44327
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:44327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:00.026340 3526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:00.026676 3526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:00.026780 3526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:00.034046 3526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:00.034170 3526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:00.034217 3526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:00.034276 3526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:00.034338 3526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:00.044386 3526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:44327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:00.045725 3526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:00.047093 3526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:00.058755 3531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:00.058710 3534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:00.058722 3532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:00.059623 3526 server_base.cc:1034] running on GCE node
I20260924 10:07:00.060338 3526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:00.061777 3526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:00.062968 3526 hybrid_clock.cc:648] HybridClock initialized: now 1790244420062915 us; error 45 us; skew 500 ppm
I20260924 10:07:00.063290 3526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:00 dist-test-slave-jvsj krb5kdc[3513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244420, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:00.066212 3526 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:07:00.067653 3526 webserver.cc:460] Webserver started at http://127.13.162.62:38933/ using document root <none> and password file <none>
I20260924 10:07:00.068281 3526 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:00.068377 3526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:00.068636 3526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:00.071609 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf"
format_stamp: "Formatted at 2026-09-24 10:07:00 on dist-test-slave-jvsj"
server_key: "b53bb88f56bcc703ca1495772eaca229"
server_key_iv: "3065634d9d81f79a41b0eb8af86dd47b"
server_key_version: "encryptionkey@0"
I20260924 10:07:00.072361 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf"
format_stamp: "Formatted at 2026-09-24 10:07:00 on dist-test-slave-jvsj"
server_key: "b53bb88f56bcc703ca1495772eaca229"
server_key_iv: "3065634d9d81f79a41b0eb8af86dd47b"
server_key_version: "encryptionkey@0"
I20260924 10:07:00.077363 3526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:07:00.080724 3541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:00.082042 3526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:07:00.082183 3526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf"
format_stamp: "Formatted at 2026-09-24 10:07:00 on dist-test-slave-jvsj"
server_key: "b53bb88f56bcc703ca1495772eaca229"
server_key_iv: "3065634d9d81f79a41b0eb8af86dd47b"
server_key_version: "encryptionkey@0"
I20260924 10:07:00.082408 3526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:00.116248 3526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:00.120566 3526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:00.120801 3526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:00.138172 3526 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:44327
I20260924 10:07:00.138192 3592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:44327 every 8 connection(s)
I20260924 10:07:00.139580 3526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:00.140277 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3526
I20260924 10:07:00.140437 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:07:00.140918 13960 external_mini_cluster.cc:1428] Setting key 9f1192a57c96ed29e03ebf5d04868803
I20260924 10:07:00.143879 3593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:07:00 dist-test-slave-jvsj krb5kdc[3513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:07:00.154382 3593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf: Bootstrap starting.
I20260924 10:07:00.157819 3593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:00.159003 3593 log.cc:824] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:00.162117 3593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf: No bootstrap required, opened a new log
I20260924 10:07:00.162642 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:00.142854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49947 (local address 127.13.162.62:44327)
0924 10:07:00.143591 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:00.143604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:00.143652 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:07:00.145284 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:00.145309 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:00.145748 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:00.146106 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:00.147536 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:00.148887 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:00.150138 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:00.150871 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:00.151047 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:00.155217 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:00.155225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:00.155239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:00.155278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:00.158371 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:00.159171 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:00.159178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:00.159185 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:00.159344 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:00.159778 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:00.159782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:00.159785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:00.160453 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:00.160670 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:00.161106 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:07:00.161454 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":316,"threads_started":1}
I20260924 10:07:00.166419 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:00.171880 3593 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44327 } }
I20260924 10:07:00.172366 3593 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:00.172474 3593 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6673ab0e8aa4818b92ebe2c64d4dfcf, State: Initialized, Role: FOLLOWER
I20260924 10:07:00.173074 3593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44327 } }
I20260924 10:07:00.173282 3593 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:00.173403 3593 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:00.173552 3593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:00.176232 3593 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44327 } }
I20260924 10:07:00.176847 3593 leader_election.cc:302] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6673ab0e8aa4818b92ebe2c64d4dfcf; no voters:
I20260924 10:07:00.177515 3593 leader_election.cc:288] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:00.177635 3603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:00.178267 3603 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [term 1 LEADER]: Becoming Leader. State: Replica: d6673ab0e8aa4818b92ebe2c64d4dfcf, State: Running, Role: LEADER
I20260924 10:07:00.178946 3603 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44327 } }
I20260924 10:07:00.181583 3593 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:00.182090 3604 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6673ab0e8aa4818b92ebe2c64d4dfcf. Latest consensus state: current_term: 1 leader_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44327 } } }
I20260924 10:07:00.182548 3604 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:00.185148 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6673ab0e8aa4818b92ebe2c64d4dfcf" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 44327 } } }
I20260924 10:07:00.185843 3608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:00.186542 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:00.187601 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:00.171340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60349 (local address 127.13.162.62:44327)
0924 10:07:00.171515 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:00.171521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:00.171977 (+ 456us) server_negotiation.cc:407] Connection header received
0924 10:07:00.172105 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:00.172110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:00.172186 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:00.172296 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:00.173833 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:00.174951 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:00.176151 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:00.176775 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:00.176906 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:00.179731 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:00.179739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:00.179745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:00.179793 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:00.183585 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:00.184403 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:00.184409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:00.184414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:00.184635 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:00.185096 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:00.185102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:00.185105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:00.185589 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:00.185889 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:00.186920 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0924 10:07:00.187161 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:07:00.193885 3608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:00.203866 3608 catalog_manager.cc:1347] Generated new cluster ID: 7337c2f259344fac8223a1f02657535a
I20260924 10:07:00.203980 3608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:00.238579 3608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:00.240051 3608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:00.252068 3608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6673ab0e8aa4818b92ebe2c64d4dfcf: Generated new TSK 0
I20260924 10:07:00.253094 3608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:07:00.269686 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3526
I20260924 10:07:00.270857 3588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:00.364701 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3526
2026-09-24T10:07:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:00 dist-test-slave-jvsj krb5kdc[3625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:00 dist-test-slave-jvsj krb5kdc[3625](info): set up 2 sockets
Sep 24 10:07:00 dist-test-slave-jvsj krb5kdc[3625](info): commencing operation
krb5kdc: starting...
W20260924 10:07:01.378633 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:07:01 dist-test-slave-jvsj krb5kdc[3625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:07:01.622637 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:46447
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:46447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:01.942912 3638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:01.943210 3638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:01.943342 3638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:01.950599 3638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:01.950721 3638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:01.950795 3638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:01.950868 3638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:01.950927 3638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:01.960799 3638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:46447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:46447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:01.962103 3638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:01.963459 3638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:01.974764 3644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:01.974967 3646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:01.975309 3643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:01.975747 3638 server_base.cc:1034] running on GCE node
I20260924 10:07:01.976537 3638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:01.977942 3638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:01.979131 3638 hybrid_clock.cc:648] HybridClock initialized: now 1790244421979072 us; error 53 us; skew 500 ppm
I20260924 10:07:01.979445 3638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:01 dist-test-slave-jvsj krb5kdc[3625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244421, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:01.982445 3638 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:07:01.983877 3638 webserver.cc:460] Webserver started at http://127.13.162.62:41991/ using document root <none> and password file <none>
I20260924 10:07:01.984539 3638 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:01.984663 3638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:01.984949 3638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:01.987988 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "b0c49722defd40939f18dcad8c4787d0"
format_stamp: "Formatted at 2026-09-24 10:07:01 on dist-test-slave-jvsj"
server_key: "c3dbb82d62c4f35cbcce3ecfb6e1bc4d"
server_key_iv: "253cbb45421c56124238152523ae3e1b"
server_key_version: "encryptionkey@0"
I20260924 10:07:01.988729 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "b0c49722defd40939f18dcad8c4787d0"
format_stamp: "Formatted at 2026-09-24 10:07:01 on dist-test-slave-jvsj"
server_key: "c3dbb82d62c4f35cbcce3ecfb6e1bc4d"
server_key_iv: "253cbb45421c56124238152523ae3e1b"
server_key_version: "encryptionkey@0"
I20260924 10:07:01.993803 3638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:07:01.997301 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:01.998767 3638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:07:01.998929 3638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "b0c49722defd40939f18dcad8c4787d0"
format_stamp: "Formatted at 2026-09-24 10:07:01 on dist-test-slave-jvsj"
server_key: "c3dbb82d62c4f35cbcce3ecfb6e1bc4d"
server_key_iv: "253cbb45421c56124238152523ae3e1b"
server_key_version: "encryptionkey@0"
I20260924 10:07:01.999123 3638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:02.014951 3638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:02.019237 3638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:02.019517 3638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:02.037035 3638 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:46447
I20260924 10:07:02.037043 3704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:46447 every 8 connection(s)
I20260924 10:07:02.038446 3638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:02.042259 3705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:02.045434 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3638
I20260924 10:07:02.045586 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:07:02.045985 13960 external_mini_cluster.cc:1428] Setting key e9f1920748eed97696e414e59ccb9667
I20260924 10:07:02.054350 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0: Bootstrap starting.
I20260924 10:07:02.058013 3705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:02.059324 3705 log.cc:824] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:02.059455 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:02.047791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46707 (local address 127.13.162.62:46447)
0924 10:07:02.048556 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:02.048571 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:02.048615 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:07:02.050359 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:02.050388 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:02.050876 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:02.051248 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:02.051865 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:02.053211 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:02.054977 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:02.055675 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:02.055818 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:02.057083 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:02.057091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:02.057106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:02.057678 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:02.057763 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:02.057791 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:02.057874 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:02.058115 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:07:02.058416 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":344,"threads_started":1}
I20260924 10:07:02.062155 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0: No bootstrap required, opened a new log
I20260924 10:07:02.063115 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:02.070154 3705 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c49722defd40939f18dcad8c4787d0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46447 } }
I20260924 10:07:02.070585 3705 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:02.070688 3705 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c49722defd40939f18dcad8c4787d0, State: Initialized, Role: FOLLOWER
I20260924 10:07:02.071295 3705 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c49722defd40939f18dcad8c4787d0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46447 } }
I20260924 10:07:02.071576 3705 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:02.071717 3705 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:02.071946 3705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:02.073221 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:02.067315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58265 (local address 127.13.162.62:46447)
0924 10:07:02.067488 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:02.067494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:02.067879 (+ 385us) server_negotiation.cc:407] Connection header received
0924 10:07:02.068060 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:02.068063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:02.068123 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:02.068206 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:02.068706 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:02.069426 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:02.070790 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:02.071250 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:02.071347 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:02.072459 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:02.072464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:02.072467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:02.072484 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:02.072528 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:02.072544 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:02.072621 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:02.072797 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 10:07:02.072936 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:07:02.074563 3705 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c49722defd40939f18dcad8c4787d0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46447 } }
I20260924 10:07:02.075084 3705 leader_election.cc:302] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c49722defd40939f18dcad8c4787d0; no voters:
I20260924 10:07:02.075623 3705 leader_election.cc:288] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:02.075736 3715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:02.076371 3715 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [term 1 LEADER]: Becoming Leader. State: Replica: b0c49722defd40939f18dcad8c4787d0, State: Running, Role: LEADER
I20260924 10:07:02.076898 3715 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c49722defd40939f18dcad8c4787d0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46447 } }
I20260924 10:07:02.077891 3705 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:02.079619 3717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c49722defd40939f18dcad8c4787d0. Latest consensus state: current_term: 1 leader_uuid: "b0c49722defd40939f18dcad8c4787d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c49722defd40939f18dcad8c4787d0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46447 } } }
I20260924 10:07:02.079594 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c49722defd40939f18dcad8c4787d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c49722defd40939f18dcad8c4787d0" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 46447 } } }
I20260924 10:07:02.080022 3717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:02.080109 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:02.081004 3721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:02.087255 3721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:02.096927 3721 catalog_manager.cc:1347] Generated new cluster ID: 6f23e4aa7da3430984f1ee617b205489
I20260924 10:07:02.097054 3721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:02.107863 3721 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:02.109139 3721 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:02.115520 3721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c49722defd40939f18dcad8c4787d0: Generated new TSK 0
I20260924 10:07:02.116561 3721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:07:02.126690 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3638
W20260924 10:07:02.127873 3700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46707
W20260924 10:07:02.128594 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3638: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:02.128729 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3638
2026-09-24T10:07:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1763 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:02 dist-test-slave-jvsj krb5kdc[3736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:02 dist-test-slave-jvsj krb5kdc[3736](info): set up 2 sockets
Sep 24 10:07:02 dist-test-slave-jvsj krb5kdc[3736](info): commencing operation
krb5kdc: starting...
W20260924 10:07:03.146603 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:07:03 dist-test-slave-jvsj krb5kdc[3736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:07:03.389014 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37523
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:03.719177 3749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:03.719518 3749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:03.719661 3749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:03.727169 3749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:03.727319 3749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:03.727403 3749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:03.727484 3749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:03.727551 3749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:03.737673 3749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:03.739074 3749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:03.740444 3749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:03.752310 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:03.752382 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:03.752382 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:03.753311 3749 server_base.cc:1034] running on GCE node
I20260924 10:07:03.754171 3749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:03.755803 3749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:03.757058 3749 hybrid_clock.cc:648] HybridClock initialized: now 1790244423757006 us; error 36 us; skew 500 ppm
I20260924 10:07:03.757505 3749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:03 dist-test-slave-jvsj krb5kdc[3736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244423, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:03.766268 3749 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:07:03.768016 3749 webserver.cc:460] Webserver started at http://127.13.162.62:41945/ using document root <none> and password file <none>
I20260924 10:07:03.768797 3749 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:03.768924 3749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:03.769224 3749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:03.772497 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "eb7546bfd55c40e5a605606862e74eda"
format_stamp: "Formatted at 2026-09-24 10:07:03 on dist-test-slave-jvsj"
server_key: "6bc2ac2ae6ffa819715dcff8bdd2f751"
server_key_iv: "c4b60bca48f9641942a619d711ed97ea"
server_key_version: "encryptionkey@0"
I20260924 10:07:03.773281 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "eb7546bfd55c40e5a605606862e74eda"
format_stamp: "Formatted at 2026-09-24 10:07:03 on dist-test-slave-jvsj"
server_key: "6bc2ac2ae6ffa819715dcff8bdd2f751"
server_key_iv: "c4b60bca48f9641942a619d711ed97ea"
server_key_version: "encryptionkey@0"
I20260924 10:07:03.778882 3749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 10:07:03.782652 3764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:03.784209 3749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:03.784397 3749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "eb7546bfd55c40e5a605606862e74eda"
format_stamp: "Formatted at 2026-09-24 10:07:03 on dist-test-slave-jvsj"
server_key: "6bc2ac2ae6ffa819715dcff8bdd2f751"
server_key_iv: "c4b60bca48f9641942a619d711ed97ea"
server_key_version: "encryptionkey@0"
I20260924 10:07:03.784596 3749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:03.790519 3749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:03.794795 3749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:03.795070 3749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:03.813256 3749 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:37523
I20260924 10:07:03.813278 3815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:37523 every 8 connection(s)
I20260924 10:07:03.814826 3749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:03.818804 3816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:03.823143 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3749
I20260924 10:07:03.823354 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:07:03.823925 13960 external_mini_cluster.cc:1428] Setting key 41e88600ccd582335b77e5d297f8dd7b
I20260924 10:07:03.831815 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:03.825838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48205 (local address 127.13.162.62:37523)
0924 10:07:03.826641 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:03.826653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:03.826695 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:07:03.828287 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:03.828311 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:03.828716 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:03.829068 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:03.830097 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:03.830105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:03.830120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:03.830135 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:03.830196 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:03.830216 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:03.830417 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:03.830531 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 10:07:03.830775 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":338,"threads_started":1}
I20260924 10:07:03.832353 3816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda: Bootstrap starting.
I20260924 10:07:03.835548 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:03.835670 3816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:03.836925 3816 log.cc:824] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:03.840596 3816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda: No bootstrap required, opened a new log
I20260924 10:07:03.843189 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:03.840713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33025 (local address 127.13.162.62:37523)
0924 10:07:03.840905 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:03.840911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:03.841147 (+ 236us) server_negotiation.cc:407] Connection header received
0924 10:07:03.841341 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:03.841344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:03.841396 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:03.841476 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:03.842569 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:03.842574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:03.842577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:03.842592 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:03.842636 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:03.842654 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:03.842748 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:03.842854 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 10:07:03.842904 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:07:03.848912 3816 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7546bfd55c40e5a605606862e74eda" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37523 } }
I20260924 10:07:03.849331 3816 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:03.849433 3816 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7546bfd55c40e5a605606862e74eda, State: Initialized, Role: FOLLOWER
I20260924 10:07:03.850059 3816 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7546bfd55c40e5a605606862e74eda" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37523 } }
I20260924 10:07:03.850299 3816 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:03.850463 3816 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:03.850623 3816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:03.852820 3816 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7546bfd55c40e5a605606862e74eda" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37523 } }
I20260924 10:07:03.853379 3816 leader_election.cc:302] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7546bfd55c40e5a605606862e74eda; no voters:
I20260924 10:07:03.853965 3816 leader_election.cc:288] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:03.854113 3826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:03.854780 3826 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [term 1 LEADER]: Becoming Leader. State: Replica: eb7546bfd55c40e5a605606862e74eda, State: Running, Role: LEADER
I20260924 10:07:03.855345 3826 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7546bfd55c40e5a605606862e74eda" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37523 } }
I20260924 10:07:03.856029 3816 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:03.862138 3827 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7546bfd55c40e5a605606862e74eda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7546bfd55c40e5a605606862e74eda" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37523 } } }
I20260924 10:07:03.863070 3827 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:03.862692 3828 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7546bfd55c40e5a605606862e74eda. Latest consensus state: current_term: 1 leader_uuid: "eb7546bfd55c40e5a605606862e74eda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7546bfd55c40e5a605606862e74eda" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37523 } } }
I20260924 10:07:03.863520 3828 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda [sys.catalog]: This master's current role is: LEADER
W20260924 10:07:03.876688 3840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:07:03.876840 3840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:07:03.877027 3841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:03.878808 3841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:03.888017 3841 catalog_manager.cc:1347] Generated new cluster ID: f320c565049949ed9f947b0545abf6a7
I20260924 10:07:03.888260 3841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:03.914654 3841 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:03.915184 3841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:03.927652 3841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7546bfd55c40e5a605606862e74eda: Generated new TSK 0
I20260924 10:07:03.928778 3841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:07:03.939608 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3749
W20260924 10:07:03.940753 3811 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48205
W20260924 10:07:03.941526 13960 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3749: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:03.941690 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3749
2026-09-24T10:07:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1812 ms)
[----------] 18 tests from AuthTokenIssuingTest (32907 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:04 dist-test-slave-jvsj krb5kdc[3847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:04 dist-test-slave-jvsj krb5kdc[3847](info): set up 2 sockets
Sep 24 10:07:04 dist-test-slave-jvsj krb5kdc[3847](info): commencing operation
krb5kdc: starting...
W20260924 10:07:04.958685 13960 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:07:05 dist-test-slave-jvsj krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:07:05.207760 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35655
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35655,127.13.162.61:42551,127.13.162.60:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:05.530766 3860 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:07:05.531104 3860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:05.531193 3860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:05.531296 3860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:05.538678 3860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:05.538776 3860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:05.538848 3860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:05.538923 3860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:05.539003 3860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:05.549032 3860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35655,127.13.162.61:42551,127.13.162.60:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:35655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:05.550359 3860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:05.551682 3860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:05.563083 3868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:05.563174 3866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:05.563144 3865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:05.563979 3860 server_base.cc:1034] running on GCE node
I20260924 10:07:05.564851 3860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:05.566305 3860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:05.567654 3860 hybrid_clock.cc:648] HybridClock initialized: now 1790244425567582 us; error 43 us; skew 500 ppm
I20260924 10:07:05.567996 3860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:05 dist-test-slave-jvsj krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:05.571063 3860 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:07:05.572463 3860 webserver.cc:460] Webserver started at http://127.13.162.62:35649/ using document root <none> and password file <none>
I20260924 10:07:05.573148 3860 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:05.573252 3860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:05.573598 3860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:05.576647 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "8eb21dab81f7410386b204dac0a2454c"
format_stamp: "Formatted at 2026-09-24 10:07:05 on dist-test-slave-jvsj"
server_key: "91c0081cf71baba51d28da286c729a39"
server_key_iv: "45e7dbbb1a0a42ac02c5963bfaa91d33"
server_key_version: "encryptionkey@0"
I20260924 10:07:05.577426 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "8eb21dab81f7410386b204dac0a2454c"
format_stamp: "Formatted at 2026-09-24 10:07:05 on dist-test-slave-jvsj"
server_key: "91c0081cf71baba51d28da286c729a39"
server_key_iv: "45e7dbbb1a0a42ac02c5963bfaa91d33"
server_key_version: "encryptionkey@0"
I20260924 10:07:05.582592 3860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:07:05.586192 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:05.587620 3860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:05.587777 3860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "8eb21dab81f7410386b204dac0a2454c"
format_stamp: "Formatted at 2026-09-24 10:07:05 on dist-test-slave-jvsj"
server_key: "91c0081cf71baba51d28da286c729a39"
server_key_iv: "45e7dbbb1a0a42ac02c5963bfaa91d33"
server_key_version: "encryptionkey@0"
I20260924 10:07:05.587992 3860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:05.609871 3860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:05.613780 3860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:05.614025 3860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:05.631266 3860 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:35655
I20260924 10:07:05.631290 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:35655 every 8 connection(s)
I20260924 10:07:05.632620 3860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:05.636301 3927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:05.639933 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3860
I20260924 10:07:05.640138 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:07:05.640733 13960 external_mini_cluster.cc:1428] Setting key bbea2236dd31818f3702f0024658b013
I20260924 10:07:05.643296 3927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:05 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.61@KRBTEST.COM" created.
I20260924 10:07:05.666977 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.646951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35655 (local address 127.0.0.1:45274)
0924 10:07:05.648121 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.648155 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:05.649953 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:05.650880 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:05.650887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:05.651611 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:05.651969 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:05.651989 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:05.653617 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:05.653621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:05.654846 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:05.654863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:05.655789 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:05.655796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:05.655960 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:05.656580 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:05.656608 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:05.658867 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:05.662985 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:05.662995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:05.663006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:05.663465 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:05.663877 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:05.663880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:05.663882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:05.664036 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:05.664835 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:05.664841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:05.665044 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 10:07:05.665272 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 10:07:05.665683 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"spinlock_wait_cycles":17664,"thread_start_us":733,"threads_started":1}
I20260924 10:07:05.667011 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.646948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.13.162.62:35655)
0924 10:07:05.647790 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:05.647807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:05.648290 (+ 483us) server_negotiation.cc:407] Connection header received
0924 10:07:05.650385 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:05.650391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:05.650643 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:05.650844 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:05.652188 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:05.653464 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:05.654985 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:05.655549 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:05.655741 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:05.659164 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:05.659176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:05.659195 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:05.659240 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:05.662843 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:05.663616 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:05.663623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:05.663626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:05.663784 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:05.664170 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:05.664175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:05.664177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:05.664719 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:05.664892 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:05.665412 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 10:07:05.665807 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
I20260924 10:07:05.670020 3927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:05.671168 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.670796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56794)
0924 10:07:05.670953 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.671060 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 10:07:05.671440 3877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
I20260924 10:07:05.673166 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.672626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56798)
0924 10:07:05.672913 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.673001 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260924 10:07:05.673362 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:42551: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab.
Entry for principal kudu/127.13.162.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab.
I20260924 10:07:05.694876 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:42551
--webserver_interface=127.13.162.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab
--principal=kudu/127.13.162.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35655,127.13.162.61:42551,127.13.162.60:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:07:05.722743 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } attempt: 1
I20260924 10:07:05.724023 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.723464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56804)
0924 10:07:05.723680 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.723832 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:07:05.725106 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.724683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56812)
0924 10:07:05.724839 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.724934 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:07:05.725296 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:42551: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
I20260924 10:07:05.793562 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } attempt: 2
I20260924 10:07:05.794806 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.794223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56822)
0924 10:07:05.794512 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.794639 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 10:07:05.795789 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.795390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56832)
0924 10:07:05.795540 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.795639 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:07:05.795984 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:42551: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
I20260924 10:07:05.887282 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } attempt: 3
I20260924 10:07:05.888540 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.888004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56836)
0924 10:07:05.888245 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.888368 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 10:07:05.889647 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:05.889223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56848)
0924 10:07:05.889390 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:05.889478 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:07:05.889833 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:42551: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
W20260924 10:07:06.016392 3934 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:07:06.016726 3934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:06.016796 3934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:06.016878 3934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:06.024152 3934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:06.024258 3934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:06.024315 3934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:06.024365 3934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:06.024425 3934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:06.033113 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } attempt: 4
I20260924 10:07:06.034396 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.033762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56852)
0924 10:07:06.034087 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.034199 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 10:07:06.034736 3934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35655,127.13.162.61:42551,127.13.162.60:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab
--principal=kudu/127.13.162.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.162.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:06.035456 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.035069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56862)
0924 10:07:06.035218 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.035316 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:07:06.035650 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:42551: Network error: Client connection negotiation failed: client connection to 127.13.162.61:42551: connect: Connection refused (error 111)
I20260924 10:07:06.036254 3934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:06.037539 3934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:06.048626 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:06.048580 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:06.048547 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:06.049518 3934 server_base.cc:1034] running on GCE node
I20260924 10:07:06.050280 3934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:06.051723 3934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:06.052930 3934 hybrid_clock.cc:648] HybridClock initialized: now 1790244426052861 us; error 45 us; skew 500 ppm
I20260924 10:07:06.053257 3934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:06.056262 3934 init.cc:377] Logged in from keytab as kudu/127.13.162.61@KRBTEST.COM (short username kudu)
I20260924 10:07:06.057776 3934 webserver.cc:460] Webserver started at http://127.13.162.61:46381/ using document root <none> and password file <none>
I20260924 10:07:06.058498 3934 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:06.058604 3934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:06.058900 3934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:06.061806 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data/instance:
uuid: "6beee305b15a4df6b4849f40761f8d99"
format_stamp: "Formatted at 2026-09-24 10:07:06 on dist-test-slave-jvsj"
server_key: "787c43200a33cb20607bdab819e3de4c"
server_key_iv: "0476e89fe3ca2218b771d8029a48257d"
server_key_version: "encryptionkey@0"
I20260924 10:07:06.062619 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal/instance:
uuid: "6beee305b15a4df6b4849f40761f8d99"
format_stamp: "Formatted at 2026-09-24 10:07:06 on dist-test-slave-jvsj"
server_key: "787c43200a33cb20607bdab819e3de4c"
server_key_iv: "0476e89fe3ca2218b771d8029a48257d"
server_key_version: "encryptionkey@0"
I20260924 10:07:06.067816 3934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:07:06.071244 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:06.072587 3934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:06.072749 3934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal
uuid: "6beee305b15a4df6b4849f40761f8d99"
format_stamp: "Formatted at 2026-09-24 10:07:06 on dist-test-slave-jvsj"
server_key: "787c43200a33cb20607bdab819e3de4c"
server_key_iv: "0476e89fe3ca2218b771d8029a48257d"
server_key_version: "encryptionkey@0"
I20260924 10:07:06.072952 3934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:06.084713 3934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:06.088889 3934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:06.089139 3934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:06.105644 3934 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.61:42551
I20260924 10:07:06.105665 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.61:42551 every 8 connection(s)
I20260924 10:07:06.107024 3934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
I20260924 10:07:06.107741 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 3934
I20260924 10:07:06.107933 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-1/wal/instance
I20260924 10:07:06.108423 13960 external_mini_cluster.cc:1428] Setting key 5256690a2019e10a4a51f09233c9f466
I20260924 10:07:06.110843 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:06.117756 4001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.162.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.60@KRBTEST.COM" created.
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:07:06.137825 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.120281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.13.162.62:35655)
0924 10:07:06.120493 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.120499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.121776 (+ 1277us) server_negotiation.cc:407] Connection header received
0924 10:07:06.123690 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.123697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.123788 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.123905 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.125398 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.126499 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.128051 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.128694 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.128863 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.132274 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.132279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.132282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.132318 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.134897 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.135604 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.135609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.135611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.135734 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.136125 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.136131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.136134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.136418 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.136548 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.137264 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.137509 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:07:06.138857 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.120815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35655 (local address 127.0.0.1:45284)
0924 10:07:06.121591 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.121642 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.123419 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.124091 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.124099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.124605 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.125204 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.125222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.126653 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.126657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.127881 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.127893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.128934 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.128939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.129152 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.129789 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.129812 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.132046 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.135034 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.135044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.135062 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.135487 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.135839 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.135844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.135848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.136021 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.136542 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.136549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.136898 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.137139 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.137679 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":317,"threads_started":1}
I20260924 10:07:06.140628 4001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:07:06.155563 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.141442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56874)
0924 10:07:06.141618 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.141636 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.141740 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.142709 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.142713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.143058 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.143206 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.143213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.144977 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.144981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.145796 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.145806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.146853 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.146858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.146994 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.147523 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.147540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.149472 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.153312 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.153315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.153318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.153651 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.154030 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.154033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.154035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.154181 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.154791 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.154796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.154917 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.155069 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.155293 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:07:06.155704 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.141504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.13.162.61:42551)
0924 10:07:06.142209 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.142217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.142247 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:07:06.142307 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.142338 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.142565 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.142719 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.143360 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.144859 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.145924 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.146715 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.146835 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.149679 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.149692 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.149710 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.149756 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.153185 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.153794 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.153803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.153807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.153944 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.154266 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.154269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.154271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.154661 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.154841 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.155202 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.155393 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":406,"threads_started":1}
I20260924 10:07:06.158046 4001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:06.159075 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.158668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48100)
0924 10:07:06.158805 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.158928 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:07:06.159284 3951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.160742 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.160429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48116)
0924 10:07:06.160537 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.160618 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab.
Entry for principal kudu/127.13.162.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab.
W20260924 10:07:06.160933 4001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.161808 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:39189
--webserver_interface=127.13.162.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab
--principal=kudu/127.13.162.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:43183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:35655,127.13.162.61:42551,127.13.162.60:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:07:06.210286 4001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 1
I20260924 10:07:06.211651 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.211063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48118)
0924 10:07:06.211308 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.211446 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 10:07:06.212635 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.212261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48124)
0924 10:07:06.212409 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.212500 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:07:06.212853 4001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.281087 4001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 2
I20260924 10:07:06.282279 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.281754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48128)
0924 10:07:06.281963 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.282079 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:07:06.283342 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.282965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48140)
0924 10:07:06.283110 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.283202 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:07:06.283568 4001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.334913 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } attempt: 5
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:07:06.347368 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.335753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.13.162.61:42551)
0924 10:07:06.335917 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.335922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.335943 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:06.336119 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.336124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.336211 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.336295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.337071 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.337770 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.338947 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.339384 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.339495 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.342509 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.342515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.342519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.342551 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.344941 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.345545 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.345550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.345554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.345673 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.346020 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.346026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.346028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.346297 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.346464 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.346840 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.347055 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:07:06.347371 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.335636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56878)
0924 10:07:06.335837 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.335858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.335993 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.336356 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.336361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.336740 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.336911 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.336920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.337900 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.337904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.338803 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.338817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.339531 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.339537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.339750 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.340333 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.340351 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.342302 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.345072 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.345078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.345082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.345438 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.345774 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.345779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.345783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.345916 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.346432 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.346437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.346602 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.346735 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.347008 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:07:06.348879 3927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:06.349906 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.349528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48142)
0924 10:07:06.349683 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.349769 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:07:06.350883 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.350545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48150)
0924 10:07:06.350683 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.350786 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:07:06.351073 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.374788 4001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 3
I20260924 10:07:06.376029 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.375421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48152)
0924 10:07:06.375671 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.375803 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:07:06.377148 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.376661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48158)
0924 10:07:06.376891 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.376988 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260924 10:07:06.377429 4001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.402294 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 1
I20260924 10:07:06.403481 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.403031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48168)
0924 10:07:06.403229 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.403322 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:07:06.404472 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.404136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48182)
0924 10:07:06.404290 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.404367 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:07:06.404664 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.472874 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 2
I20260924 10:07:06.473958 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.473466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48196)
0924 10:07:06.473688 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.473794 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 10:07:06.474947 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.474589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48208)
0924 10:07:06.474756 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.474831 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:07:06.475132 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
W20260924 10:07:06.483402 4008 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:07:06.483731 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:06.483824 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:06.483933 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:06.491084 4008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:06.491185 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:06.491274 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:06.491338 4008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:06.491417 4008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:06.501358 4008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:43183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:35655,127.13.162.61:42551,127.13.162.60:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab
--principal=kudu/127.13.162.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.162.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:06.502763 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:06.504098 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:06.517117 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:06.517061 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:06.517154 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:06.517926 4008 server_base.cc:1034] running on GCE node
I20260924 10:07:06.518723 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:06.520152 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:06.520678 4001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 4
I20260924 10:07:06.521389 4008 hybrid_clock.cc:648] HybridClock initialized: now 1790244426521308 us; error 43 us; skew 500 ppm
I20260924 10:07:06.521759 4008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:07:06.521791 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.521308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48216)
0924 10:07:06.521529 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.521614 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:07:06.522783 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.522469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48220)
0924 10:07:06.522607 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.522676 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:07:06.522986 4001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:06.525034 4008 init.cc:377] Logged in from keytab as kudu/127.13.162.60@KRBTEST.COM (short username kudu)
I20260924 10:07:06.526439 4008 webserver.cc:460] Webserver started at http://127.13.162.60:37731/ using document root <none> and password file <none>
I20260924 10:07:06.527110 4008 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:06.527195 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:06.527464 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:06.530444 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data/instance:
uuid: "f8dd13e5f2694b41acdbff574ddc21fa"
format_stamp: "Formatted at 2026-09-24 10:07:06 on dist-test-slave-jvsj"
server_key: "18cd20c5438920a8dc6ceab5eccbab6c"
server_key_iv: "abda94f96635a0050539eb7eae764adc"
server_key_version: "encryptionkey@0"
I20260924 10:07:06.531190 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal/instance:
uuid: "f8dd13e5f2694b41acdbff574ddc21fa"
format_stamp: "Formatted at 2026-09-24 10:07:06 on dist-test-slave-jvsj"
server_key: "18cd20c5438920a8dc6ceab5eccbab6c"
server_key_iv: "abda94f96635a0050539eb7eae764adc"
server_key_version: "encryptionkey@0"
I20260924 10:07:06.536240 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:06.539564 4023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:06.540886 4008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:06.541043 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal
uuid: "f8dd13e5f2694b41acdbff574ddc21fa"
format_stamp: "Formatted at 2026-09-24 10:07:06 on dist-test-slave-jvsj"
server_key: "18cd20c5438920a8dc6ceab5eccbab6c"
server_key_iv: "abda94f96635a0050539eb7eae764adc"
server_key_version: "encryptionkey@0"
I20260924 10:07:06.541240 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:06.572970 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:06.577345 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:06.577602 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:06.581347 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 3
I20260924 10:07:06.582576 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.582012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48228)
0924 10:07:06.582249 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.582385 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:07:06.583642 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.583301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48230)
0924 10:07:06.583464 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.583535 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 10:07:06.583899 3927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:39189: Network error: Client connection negotiation failed: client connection to 127.13.162.60:39189: connect: Connection refused (error 111)
I20260924 10:07:06.595136 4008 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.60:39189
I20260924 10:07:06.595157 4074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.60:39189 every 8 connection(s)
I20260924 10:07:06.596481 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
I20260924 10:07:06.600034 4075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:06.604722 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 4008
I20260924 10:07:06.604904 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244085964582-13960-0/minicluster-data/master-2/wal/instance
I20260924 10:07:06.605413 13960 external_mini_cluster.cc:1428] Setting key 32e70aef69a30a82f646c09fc6e18146
I20260924 10:07:06.607093 4075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
I20260924 10:07:06.624766 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.609755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.13.162.62:35655)
0924 10:07:06.609973 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.609979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.610490 (+ 511us) server_negotiation.cc:407] Connection header received
0924 10:07:06.610750 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.610756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.610830 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.610939 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.612148 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.612830 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.614434 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.614935 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.615026 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.618589 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.618594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.618597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.618630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.621637 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.622423 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.622427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.622430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.622559 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.623014 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.623019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.623022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.623339 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.623467 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.624164 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.624387 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:07:06.625748 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.607319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.13.162.60:39189)
0924 10:07:06.608091 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.608106 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.608165 (+ 59us) server_negotiation.cc:407] Connection header received
0924 10:07:06.609919 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.609952 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.610504 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.610799 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.612115 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.613405 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.614664 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.615289 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.615400 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.619129 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.619138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.619150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.619178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.622221 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.622937 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.622943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.622945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.623097 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.623458 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.623462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.623465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.623840 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.624097 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.624423 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.624672 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":563,"spinlock_wait_cycles":14080,"thread_start_us":351,"threads_started":1}
I20260924 10:07:06.625787 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.609704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:35655 (local address 127.0.0.1:45298)
0924 10:07:06.610349 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.610388 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.610549 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.611010 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.611016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.611632 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.611986 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.612003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.612978 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.612983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.614272 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.614284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.615140 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.615147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.615381 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.616128 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.616158 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.618396 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.621763 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.621773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.621782 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.622299 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.622663 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.622668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.622671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.622905 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.623461 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.623468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.623801 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.624040 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.624557 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":3,"spinlock_wait_cycles":18176,"thread_start_us":267,"threads_started":1}
I20260924 10:07:06.627895 4075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:07:06.641909 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.628599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:42551 (local address 127.0.0.1:56888)
0924 10:07:06.628761 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.628779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.628853 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.629197 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.629202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.629643 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.629781 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.629789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.631138 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.631144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.632092 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.632107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.633028 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.633034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.633226 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.633796 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.633818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.636197 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.639584 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.639589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.639592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.639968 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.640414 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.640418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.640420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.640616 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.641224 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.641227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.641351 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.641463 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.641666 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:07:06.642179 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.628706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.13.162.61:42551)
0924 10:07:06.628872 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.628878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.628896 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:07:06.628955 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.628960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.629038 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.629135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.629932 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.631004 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.632248 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.632887 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.632998 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.636409 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.636415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.636420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.636462 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.639455 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.640088 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.640093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.640097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.640271 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.640737 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.640742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.640745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.641084 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.641245 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.641588 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.641821 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:07:06.643204 4075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:06.645524 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.628704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.13.162.62:35655)
0924 10:07:06.628860 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.628865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.629108 (+ 243us) server_negotiation.cc:407] Connection header received
0924 10:07:06.629243 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.629248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.629317 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.629416 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.630675 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.631795 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.633118 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.633648 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.633767 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.638848 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.638854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.638858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.638894 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.642029 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.642845 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.642851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.642855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.643012 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.643384 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.643389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.643392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.643743 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.644329 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.644863 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.645099 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:07:06.656384 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.643841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48248)
0924 10:07:06.643989 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.644012 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.644106 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.644488 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.644492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.644938 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.645123 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.645135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.646474 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.646480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.647520 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.647532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.648138 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.648144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.648369 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.648954 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.648971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.650870 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.654048 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.654053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.654055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.654435 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.654771 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.654775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.654778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.654917 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.655393 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.655397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.655536 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.655671 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.655986 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260924 10:07:06.656389 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.644115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.13.162.60:39189)
0924 10:07:06.644255 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.644260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.644274 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:07:06.644310 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.644313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.644361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.644431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.645420 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.646355 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.647644 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.648015 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.648126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.651035 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.651040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.651043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.651071 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.653930 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.654535 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.654540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.654544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.654664 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.655009 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.655014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.655018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.655289 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.655410 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.655776 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.656104 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":45}
I20260924 10:07:06.660866 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.647124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.13.162.61:42551)
0924 10:07:06.647445 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.647449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.647462 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:07:06.647499 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.647502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.647552 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.647626 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.648790 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.649503 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.650862 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.651339 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.651429 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.655661 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.655668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.655672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.655717 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.658475 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.659073 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.659078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.659082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.659216 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.659558 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.659561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.659563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.659827 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.660080 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.660358 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.660580 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260924 10:07:06.661180 4075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: Bootstrap starting.
I20260924 10:07:06.664821 4075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:06.665328 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:06.665887 4075 log.cc:824] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:06.668565 4075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: No bootstrap required, opened a new log
I20260924 10:07:06.668061 3902 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8eb21dab81f7410386b204dac0a2454c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45308
I20260924 10:07:06.675510 4075 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.675822 4075 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8dd13e5f2694b41acdbff574ddc21fa, State: Initialized, Role: FOLLOWER
I20260924 10:07:06.676239 4075 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.677142 4081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.677506 4081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:06.678361 4075 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:07:06.686962 4092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:07:06.687090 4092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:07:06.685138 4008 master_runner.cc:186] Error getting master registration for 127.13.162.62:35655: OK, instance_id { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" instance_seqno: 1790244425626393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:07:06.692885 4008 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:07:06.761168 3927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 4
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244425, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
I20260924 10:07:06.776525 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.762164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.13.162.60:39189)
0924 10:07:06.762357 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.762363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.762403 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:07:06.762462 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.762466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.762534 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.762630 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.763450 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.764168 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.765445 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.765926 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.766042 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.769535 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.769542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.769548 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.769589 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.773295 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.774158 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.774164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.774177 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.774368 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.774774 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.774779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.774783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.775261 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.775411 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.775853 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.776086 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:07:06.776535 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.761895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48262)
0924 10:07:06.762127 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.762149 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.762286 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.762700 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.762705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.763113 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.763282 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.763293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.764328 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.764335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.765286 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.765302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.766075 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.766081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.766288 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.766928 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.766947 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.769276 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.773450 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.773456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.773461 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.774027 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.774476 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.774481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.774485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.774667 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.775391 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.775396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.775586 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.775754 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.776057 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:07:06.782981 3927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: Bootstrap starting.
I20260924 10:07:06.787063 3927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:06.788357 3927 log.cc:824] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:06.791697 3927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: No bootstrap required, opened a new log
I20260924 10:07:06.799347 3927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.799800 3927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb21dab81f7410386b204dac0a2454c, State: Initialized, Role: FOLLOWER
I20260924 10:07:06.800318 3927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.801330 4095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.801697 4095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:06.802656 3927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:06.802750 3902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:07:06.802948 3902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:06.805356 3902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.806690 3902 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [CANDIDATE]: Term 1 election: Requested vote from peers 6beee305b15a4df6b4849f40761f8d99 (127.13.162.61:42551), f8dd13e5f2694b41acdbff574ddc21fa (127.13.162.60:39189)
I20260924 10:07:06.807974 4050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8eb21dab81f7410386b204dac0a2454c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8dd13e5f2694b41acdbff574ddc21fa"
I20260924 10:07:06.808472 4050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:06.811937 4050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb21dab81f7410386b204dac0a2454c in term 1.
I20260924 10:07:06.812668 3877 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eb21dab81f7410386b204dac0a2454c, f8dd13e5f2694b41acdbff574ddc21fa; no voters:
I20260924 10:07:06.813201 4095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:06.818060 4095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [term 1 LEADER]: Becoming Leader. State: Replica: 8eb21dab81f7410386b204dac0a2454c, State: Running, Role: LEADER
I20260924 10:07:06.818799 4095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.822154 4001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } attempt: 5
W20260924 10:07:06.829640 4029 tablet.cc:2369] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:07:06 dist-test-slave-jvsj krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244426, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
I20260924 10:07:06.847929 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.823752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.13.162.60:39189)
0924 10:07:06.823946 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.823953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.823973 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:07:06.824041 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.824046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.824152 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.824265 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.825485 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.826734 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.828652 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.829392 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.829507 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.834712 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.834720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.834726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.834777 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.838910 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.840334 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.840340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.840369 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.840648 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.841901 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.841906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.841910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.842964 (+ 1054us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.844060 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.844957 (+ 897us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.846496 (+ 1539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:07:06.848007 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.822871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:39189 (local address 127.0.0.1:48264)
0924 10:07:06.823312 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:06.823334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:06.823475 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:06.824479 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:06.824485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:06.825038 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:06.825280 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.825294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.826922 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.826928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.828455 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:06.828471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.829782 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.829788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:06.830009 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.830886 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:06.830914 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:06.834378 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:06.839208 (+ 4830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.839214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.839218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.840090 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.840922 (+ 832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:06.840928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:06.840933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:06.841560 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:06.843246 (+ 1686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:06.843251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:06.843768 (+ 517us) client_negotiation.cc:769] Sending connection context
0924 10:07:06.844793 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0924 10:07:06.846673 (+ 1880us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260924 10:07:06.855520 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: Bootstrap starting.
I20260924 10:07:06.861680 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: Neither blocks nor log segments found. Creating new log.
W20260924 10:07:06.866819 3881 tablet.cc:2369] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:06.866732 4099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb21dab81f7410386b204dac0a2454c. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.867214 4099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:06.867051 4001 log.cc:824] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:06.880451 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: No bootstrap required, opened a new log
W20260924 10:07:06.899612 4113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:07:06.899768 4113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:07:06.899907 4109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:06.902165 4109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:06.950290 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.887064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.13.162.62:35655)
0924 10:07:06.890568 (+ 3504us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.890575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.917629 (+ 27054us) server_negotiation.cc:407] Connection header received
0924 10:07:06.919043 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.919052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.919164 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.919247 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.929133 (+ 9886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.930302 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.933302 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.934001 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.934108 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.938926 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.938933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.938938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.938982 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.942626 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.944123 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.944129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.944132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.944300 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.945484 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.945490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.945493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.945877 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.946122 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.949663 (+ 3541us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.949928 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3333}
I20260924 10:07:06.886991 3860 master_runner.cc:186] Error getting master registration for 127.13.162.61:42551: OK, instance_id { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" instance_seqno: 1790244426101021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:07:06.954798 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.955189 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6beee305b15a4df6b4849f40761f8d99, State: Initialized, Role: FOLLOWER
I20260924 10:07:06.955673 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } }
I20260924 10:07:06.956437 3860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:07:06.958540 4050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [term 1 FOLLOWER]: Refusing update from remote peer 8eb21dab81f7410386b204dac0a2454c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:06.959563 4099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:06.962836 4120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.964331 4120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:06.968106 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.891604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.13.162.61:42551)
0924 10:07:06.891780 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.891787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.942966 (+ 51179us) server_negotiation.cc:407] Connection header received
0924 10:07:06.943427 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.943433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.943521 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.943587 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.945911 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.947174 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.956221 (+ 9047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.956843 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.956939 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.960177 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.960183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.960187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.960226 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.963817 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.965204 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.965209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.965213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.965367 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.966033 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.966038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.966042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.966463 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.966701 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.967516 (+ 815us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.967759 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:07:06.971066 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:06.972525 3975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:06.976220 3975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [term 1 FOLLOWER]: Refusing update from remote peer 8eb21dab81f7410386b204dac0a2454c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:06.971916 3976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8eb21dab81f7410386b204dac0a2454c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6beee305b15a4df6b4849f40761f8d99"
I20260924 10:07:06.977229 4099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:06.980700 3976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb21dab81f7410386b204dac0a2454c in term 1.
I20260924 10:07:06.985572 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:06.914564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.13.162.60:39189)
0924 10:07:06.914783 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:06.914790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:06.948372 (+ 33582us) server_negotiation.cc:407] Connection header received
0924 10:07:06.948717 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:06.948724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:06.948836 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:06.948906 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:06.956666 (+ 7760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.957737 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.970780 (+ 13043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:06.971528 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:06.971678 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:06.978096 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:06.978107 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:06.978113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:06.978156 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:06.982164 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.983006 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.983013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.983016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.983196 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:06.983699 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:06.983704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:06.983708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:06.984063 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:06.984314 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:06.984860 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 10:07:06.985162 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:07:06.991528 4081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb21dab81f7410386b204dac0a2454c. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.991856 4081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:06.993605 4119 mvcc.cc:205] Tried to move back new op lower bound from 7332841172805148672 to 7332841172260028416. Current Snapshot: MvccSnapshot[applied={T|T < 7332841172805148672}]
I20260924 10:07:06.997262 4099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.997562 4099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:06.998735 4081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:06.998989 4081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:06.999248 4121 mvcc.cc:205] Tried to move back new op lower bound from 7332841172805148672 to 7332841172260028416. Current Snapshot: MvccSnapshot[applied={T|T < 7332841172805148672}]
I20260924 10:07:07.001645 4109 catalog_manager.cc:1347] Generated new cluster ID: ec848aab0f2c4ed7b3537cada167fa42
I20260924 10:07:07.001749 4109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:07.014499 4120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb21dab81f7410386b204dac0a2454c. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:07.014842 4120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:07.018618 4095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:07.018990 4095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:07.025136 4125 mvcc.cc:205] Tried to move back new op lower bound from 7332841172805148672 to 7332841172260028416. Current Snapshot: MvccSnapshot[applied={T|T < 7332841172805148672}]
I20260924 10:07:07.035810 4120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8eb21dab81f7410386b204dac0a2454c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb21dab81f7410386b204dac0a2454c" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 35655 } } peers { permanent_uuid: "6beee305b15a4df6b4849f40761f8d99" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 42551 } } peers { permanent_uuid: "f8dd13e5f2694b41acdbff574ddc21fa" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 39189 } } }
I20260924 10:07:07.036111 4120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:07.052958 4138 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:07:07.072491 4109 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:07.074816 4109 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:07.101812 4109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eb21dab81f7410386b204dac0a2454c: Generated new TSK 0
I20260924 10:07:07.103495 4109 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:07:07.191381 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.188520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.13.162.61:42551)
0924 10:07:07.188731 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.188738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.189333 (+ 595us) server_negotiation.cc:407] Connection header received
0924 10:07:07.189621 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.189627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.189725 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.189817 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.191119 (+ 1302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56930: BlockingRecv error: recv got EOF from 127.0.0.1:56930 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 10:07:07.191572 3929 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.188289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.13.162.62:35655)
0924 10:07:07.188519 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.188526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.189543 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 10:07:07.189786 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.189793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.189874 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.189942 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.191325 (+ 1383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45334: BlockingRecv error: recv got EOF from 127.0.0.1:45334 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 10:07:07.193688 4078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.190173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.13.162.60:39189)
0924 10:07:07.190383 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.190389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.192339 (+ 1950us) server_negotiation.cc:407] Connection header received
0924 10:07:07.192600 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.192605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.192687 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.192774 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.193482 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48274: BlockingRecv error: recv got EOF from 127.0.0.1:48274 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:07:07.200042 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.197295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.13.162.61:42551)
0924 10:07:07.197428 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.197432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.198705 (+ 1273us) server_negotiation.cc:407] Connection header received
0924 10:07:07.198908 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.198913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.199009 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.199093 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.199777 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56932: BlockingRecv error: recv got EOF from 127.0.0.1:56932 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:07:07.205470 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.203965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.13.162.61:42551)
0924 10:07:07.204115 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.204120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.204323 (+ 203us) server_negotiation.cc:407] Connection header received
0924 10:07:07.204492 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.204497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.204562 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.204640 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.205280 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56944: BlockingRecv error: recv got EOF from 127.0.0.1:56944 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:07:07.212431 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.210803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.13.162.61:42551)
0924 10:07:07.210974 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.210979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.211250 (+ 271us) server_negotiation.cc:407] Connection header received
0924 10:07:07.211403 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.211407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.211468 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.211542 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.212217 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56956: BlockingRecv error: recv got EOF from 127.0.0.1:56956 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 10:07:07.221310 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.219561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.13.162.61:42551)
0924 10:07:07.219762 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.219768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.220034 (+ 266us) server_negotiation.cc:407] Connection header received
0924 10:07:07.220226 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.220231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.220304 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.220384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.221108 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56964: BlockingRecv error: recv got EOF from 127.0.0.1:56964 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:07:07.234243 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.232579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.13.162.61:42551)
0924 10:07:07.232768 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.232775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.233084 (+ 309us) server_negotiation.cc:407] Connection header received
0924 10:07:07.233221 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.233226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.233310 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.233418 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.234014 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56970: BlockingRecv error: recv got EOF from 127.0.0.1:56970 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 10:07:07.255395 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.253488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.13.162.61:42551)
0924 10:07:07.253699 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.253705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.254070 (+ 365us) server_negotiation.cc:407] Connection header received
0924 10:07:07.254224 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.254229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.254344 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.254471 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.255154 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56978: BlockingRecv error: recv got EOF from 127.0.0.1:56978 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:07:07.297716 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.295672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.13.162.61:42551)
0924 10:07:07.295893 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.295898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.296250 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:07:07.296459 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.296463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.296535 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.296619 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.297438 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56986: BlockingRecv error: recv got EOF from 127.0.0.1:56986 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 10:07:07.367364 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.365434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.13.162.61:42551)
0924 10:07:07.365631 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.365638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.366029 (+ 391us) server_negotiation.cc:407] Connection header received
0924 10:07:07.366205 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.366212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.366308 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.366457 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.367133 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56998: BlockingRecv error: recv got EOF from 127.0.0.1:56998 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 10:07:07.501493 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.499643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.13.162.61:42551)
0924 10:07:07.499873 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.499879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.500168 (+ 289us) server_negotiation.cc:407] Connection header received
0924 10:07:07.500356 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.500360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.500435 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.500520 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.501236 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57006: BlockingRecv error: recv got EOF from 127.0.0.1:57006 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:07:07.688993 4092 catalog_manager.cc:1259] Loaded cluster ID: ec848aab0f2c4ed7b3537cada167fa42
I20260924 10:07:07.689095 4092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: loading cluster ID for follower catalog manager: success
I20260924 10:07:07.691572 4092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: acquiring CA information for follower catalog manager: success
I20260924 10:07:07.693392 4092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8dd13e5f2694b41acdbff574ddc21fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:07:07.763036 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:07.761456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.13.162.61:42551)
0924 10:07:07.761658 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:07.761662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:07.761847 (+ 185us) server_negotiation.cc:407] Connection header received
0924 10:07:07.762013 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:07.762017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:07.762101 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:07.762196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:07.762872 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57020: BlockingRecv error: recv got EOF from 127.0.0.1:57020 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:07:08.057336 4138 catalog_manager.cc:1259] Loaded cluster ID: ec848aab0f2c4ed7b3537cada167fa42
I20260924 10:07:08.057442 4138 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: loading cluster ID for follower catalog manager: success
I20260924 10:07:08.060199 4138 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: acquiring CA information for follower catalog manager: success
I20260924 10:07:08.062106 4138 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6beee305b15a4df6b4849f40761f8d99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:07:08.285940 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:08.278753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.13.162.61:42551)
0924 10:07:08.279859 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:08.279867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:08.280396 (+ 529us) server_negotiation.cc:407] Connection header received
0924 10:07:08.280618 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:08.280623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:08.280773 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:08.280884 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:08.281342 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.282222 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.283583 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.284040 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.284165 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:08.284673 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:08.285280 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:08.285439 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:08.285518 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:07:08.285608 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":706,"threads_started":1}
I20260924 10:07:08.304121 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:08.292044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.13.162.60:39189)
0924 10:07:08.298721 (+ 6677us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:08.298726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:08.298747 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:08.298852 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:08.298856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:08.299007 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:08.299167 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:08.299731 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.300429 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.301770 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.302143 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.302258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:08.302821 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:08.303358 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:08.303493 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:08.303752 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 10:07:08.303825 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6558,"thread_start_us":6448,"threads_started":1}
I20260924 10:07:08.330467 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:08.310406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.13.162.62:35655)
0924 10:07:08.322839 (+ 12433us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:08.322846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:08.322871 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:07:08.323012 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:08.323020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:08.323261 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:08.323456 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:08.324348 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.325041 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.327139 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.327770 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.327891 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:08.328732 (+ 841us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:08.329167 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:08.329268 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:08.330029 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 10:07:08.330119 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12293,"thread_start_us":13168,"threads_started":1}
I20260924 10:07:08.332122 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:08.310543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.13.162.61:42551)
0924 10:07:08.310753 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:08.310760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:08.324262 (+ 13502us) server_negotiation.cc:407] Connection header received
0924 10:07:08.325432 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:08.325437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:08.325531 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:08.325591 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:08.326402 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.327480 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.329495 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.330127 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.330234 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:08.331298 (+ 1064us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:08.331425 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:08.331488 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:08.331706 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:07:08.331808 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:07:08.335932 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3860
I20260924 10:07:08.337085 3922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:08.338582 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:08.317573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.13.162.60:39189)
0924 10:07:08.317782 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:08.317789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:08.330312 (+ 12523us) server_negotiation.cc:407] Connection header received
0924 10:07:08.330431 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:08.330436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:08.330538 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:08.330623 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:08.332999 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.334035 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.336641 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:08.337108 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:08.337194 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:08.337794 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:08.337924 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:08.337986 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:08.338196 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 10:07:08.338282 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:07:08.448531 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3860
I20260924 10:07:08.467873 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 3934
I20260924 10:07:08.469154 3996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:08.573489 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 3934
I20260924 10:07:08.590204 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 4008
I20260924 10:07:08.591297 4070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:08.696703 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 4008
2026-09-24T10:07:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4759 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:08 dist-test-slave-jvsj krb5kdc[4227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:08 dist-test-slave-jvsj krb5kdc[4227](info): set up 2 sockets
Sep 24 10:07:08 dist-test-slave-jvsj krb5kdc[4227](info): commencing operation
krb5kdc: starting...
W20260924 10:07:09.738740 13960 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/test-user.keytab.
Sep 24 10:07:09 dist-test-slave-jvsj krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.162.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
Entry for principal kudu/127.13.162.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab.
I20260924 10:07:09.989450 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37453
--webserver_interface=127.13.162.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37453,127.13.162.61:34601,127.13.162.60:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:10.326498 4240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:07:10.326880 4240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:10.326980 4240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:10.327092 4240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:10.334503 4240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:10.334617 4240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:10.334712 4240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:10.334779 4240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:10.334865 4240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:10.345077 4240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37453,127.13.162.61:34601,127.13.162.60:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.62.keytab
--principal=kudu/127.13.162.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.62:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.162.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:10.346500 4240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:10.347846 4240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:10.359495 4245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:10.359460 4248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:10.359433 4246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:10.360270 4240 server_base.cc:1034] running on GCE node
I20260924 10:07:10.361110 4240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:10.362639 4240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:10.363910 4240 hybrid_clock.cc:648] HybridClock initialized: now 1790244430363848 us; error 61 us; skew 500 ppm
I20260924 10:07:10.364272 4240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:10 dist-test-slave-jvsj krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:10.367399 4240 init.cc:377] Logged in from keytab as kudu/127.13.162.62@KRBTEST.COM (short username kudu)
I20260924 10:07:10.368876 4240 webserver.cc:460] Webserver started at http://127.13.162.62:40223/ using document root <none> and password file <none>
I20260924 10:07:10.369553 4240 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:10.369652 4240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:10.369978 4240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:10.373093 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data/instance:
uuid: "905c660cd9da4df186ac5c4d6f691bfa"
format_stamp: "Formatted at 2026-09-24 10:07:10 on dist-test-slave-jvsj"
server_key: "ead863e96f6e88687d0095f2ac602278"
server_key_iv: "cf5fe20d0d648a0526aef3851a2a7482"
server_key_version: "encryptionkey@0"
I20260924 10:07:10.373905 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal/instance:
uuid: "905c660cd9da4df186ac5c4d6f691bfa"
format_stamp: "Formatted at 2026-09-24 10:07:10 on dist-test-slave-jvsj"
server_key: "ead863e96f6e88687d0095f2ac602278"
server_key_iv: "cf5fe20d0d648a0526aef3851a2a7482"
server_key_version: "encryptionkey@0"
I20260924 10:07:10.379357 4240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 10:07:10.382984 4255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:10.384488 4240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:10.384646 4240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal
uuid: "905c660cd9da4df186ac5c4d6f691bfa"
format_stamp: "Formatted at 2026-09-24 10:07:10 on dist-test-slave-jvsj"
server_key: "ead863e96f6e88687d0095f2ac602278"
server_key_iv: "cf5fe20d0d648a0526aef3851a2a7482"
server_key_version: "encryptionkey@0"
I20260924 10:07:10.384872 4240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:10.399732 4240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:10.404202 4240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:10.404479 4240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:10.423641 4240 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.62:37453
I20260924 10:07:10.423676 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.62:37453 every 8 connection(s)
I20260924 10:07:10.425112 4240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:10.429031 4307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:10.434409 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 4240
I20260924 10:07:10.434602 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-0/wal/instance
I20260924 10:07:10.435093 13960 external_mini_cluster.cc:1428] Setting key c0f249c34544a242572abfd8864a0852
I20260924 10:07:10.436584 4307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:10 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.61@KRBTEST.COM; defaulting to no policy
I20260924 10:07:10.461603 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.440500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.13.162.62:37453)
0924 10:07:10.442389 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:10.442396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:10.442444 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:07:10.444328 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:10.444335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:10.444600 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:10.444775 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:10.446010 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.447790 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.449195 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.449991 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.450176 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:10.453540 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:10.453551 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:10.453569 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:10.453604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:10.457087 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:10.457837 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:10.457847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:10.457851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:10.458058 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:10.458540 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:10.458545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:10.458548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:10.459285 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:10.459502 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:10.460028 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 10:07:10.460450 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":1657,"spinlock_wait_cycles":92032,"thread_start_us":1423,"threads_started":1}
I20260924 10:07:10.461649 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.440500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37453 (local address 127.0.0.1:34974)
0924 10:07:10.441455 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.441512 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:10.443895 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:10.444874 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:10.444881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:10.445452 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:10.445795 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:10.445815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.447935 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.447939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:10.449051 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:10.449064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.450212 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.450218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:10.450396 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:10.451027 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:10.451049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:10.453278 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:10.457237 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:10.457248 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:10.457259 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:10.457694 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:10.458170 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:10.458173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:10.458175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:10.458434 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:10.459412 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:10.459419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:10.459600 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 10:07:10.459829 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 10:07:10.460268 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":412,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.61@KRBTEST.COM" created.
I20260924 10:07:10.465361 4307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:10.466686 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.466111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45442)
0924 10:07:10.466363 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.466529 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:07:10.467298 4257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
I20260924 10:07:10.468906 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.468560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45456)
0924 10:07:10.468703 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.468777 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:07:10.469090 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:34601: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab.
Entry for principal kudu/127.13.162.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab.
I20260924 10:07:10.493208 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:34601
--webserver_interface=127.13.162.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab
--principal=kudu/127.13.162.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37453,127.13.162.61:34601,127.13.162.60:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:07:10.518453 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } attempt: 1
I20260924 10:07:10.519516 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.519089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45472)
0924 10:07:10.519271 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.519379 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:07:10.520498 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.520191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45486)
0924 10:07:10.520306 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.520372 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 10:07:10.520663 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:34601: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
I20260924 10:07:10.588891 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } attempt: 2
I20260924 10:07:10.590297 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.589651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45492)
0924 10:07:10.589980 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.590093 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:07:10.591408 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.591051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45498)
0924 10:07:10.591200 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.591284 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:07:10.591593 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:34601: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
I20260924 10:07:10.682858 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } attempt: 3
I20260924 10:07:10.684057 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.683605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45514)
0924 10:07:10.683831 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.683923 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 10:07:10.685091 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.684763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:45520)
0924 10:07:10.684884 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.684953 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:07:10.685299 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:34601: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
W20260924 10:07:10.820672 4314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:07:10.821038 4314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:10.821146 4314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:10.821262 4314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:07:10.828516 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } attempt: 4
W20260924 10:07:10.829407 4314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:10.829521 4314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:10.829615 4314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:10.829674 4314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:10.829756 4314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:10.829843 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.829220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:43332)
0924 10:07:10.829530 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.829635 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260924 10:07:10.830874 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.830528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:43336)
0924 10:07:10.830645 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.830729 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 10:07:10.831034 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.61:34601: Network error: Client connection negotiation failed: client connection to 127.13.162.61:34601: connect: Connection refused (error 111)
I20260924 10:07:10.840452 4314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37453,127.13.162.61:34601,127.13.162.60:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.61.keytab
--principal=kudu/127.13.162.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.61:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.162.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:10.841734 4314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:10.843058 4314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:10.855080 4320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:10.855032 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:10.855108 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:10.859912 4314 server_base.cc:1034] running on GCE node
I20260924 10:07:10.860801 4314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:10.862342 4314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:10.863581 4314 hybrid_clock.cc:648] HybridClock initialized: now 1790244430863513 us; error 48 us; skew 500 ppm
I20260924 10:07:10.863917 4314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:10 dist-test-slave-jvsj krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:10.867111 4314 init.cc:377] Logged in from keytab as kudu/127.13.162.61@KRBTEST.COM (short username kudu)
I20260924 10:07:10.868649 4314 webserver.cc:460] Webserver started at http://127.13.162.61:33461/ using document root <none> and password file <none>
I20260924 10:07:10.869378 4314 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:10.869480 4314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:10.869786 4314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:10.872918 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data/instance:
uuid: "9bc3d0ff39f44bb2a732b8801f98851b"
format_stamp: "Formatted at 2026-09-24 10:07:10 on dist-test-slave-jvsj"
server_key: "0775830b16a4f235eff67e81ac376a1e"
server_key_iv: "9160e0689595c60770e672d9fe35d7e0"
server_key_version: "encryptionkey@0"
I20260924 10:07:10.873726 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal/instance:
uuid: "9bc3d0ff39f44bb2a732b8801f98851b"
format_stamp: "Formatted at 2026-09-24 10:07:10 on dist-test-slave-jvsj"
server_key: "0775830b16a4f235eff67e81ac376a1e"
server_key_iv: "9160e0689595c60770e672d9fe35d7e0"
server_key_version: "encryptionkey@0"
I20260924 10:07:10.879101 4314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 10:07:10.882807 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:10.884265 4314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:07:10.884423 4314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal
uuid: "9bc3d0ff39f44bb2a732b8801f98851b"
format_stamp: "Formatted at 2026-09-24 10:07:10 on dist-test-slave-jvsj"
server_key: "0775830b16a4f235eff67e81ac376a1e"
server_key_iv: "9160e0689595c60770e672d9fe35d7e0"
server_key_version: "encryptionkey@0"
I20260924 10:07:10.884610 4314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:10.904654 4314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:10.908896 4314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:10.909121 4314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:10.926808 4314 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.61:34601
I20260924 10:07:10.926839 4380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.61:34601 every 8 connection(s)
I20260924 10:07:10.928215 4314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/data/info.pb
I20260924 10:07:10.932070 4381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:10.936929 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 4314
I20260924 10:07:10.937091 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-1/wal/instance
I20260924 10:07:10.937501 13960 external_mini_cluster.cc:1428] Setting key 2d5fa9213c8ed81fc5dc54ab861d4034
I20260924 10:07:10.938972 4381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:10 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.162.60@KRBTEST.COM; defaulting to no policy
I20260924 10:07:10.960925 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.941712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.13.162.62:37453)
0924 10:07:10.941931 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:10.941937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:10.943118 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 10:07:10.945117 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:10.945125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:10.945223 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:10.945341 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:10.946996 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.948015 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.949584 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.950155 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.950280 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:10.954261 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:10.954268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:10.954273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:10.954345 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:10.957733 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:10.958561 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:10.958576 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:10.958580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:10.958733 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:10.959131 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:10.959137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:10.959140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:10.959522 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:10.959702 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:10.960419 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 10:07:10.960667 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:07:10.962005 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.942125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37453 (local address 127.0.0.1:53856)
0924 10:07:10.942948 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.942994 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:10.944823 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:10.945539 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:10.945547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:10.946122 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:10.946773 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:10.946794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.948193 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.948197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:10.949438 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:10.949451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.950498 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.950507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:10.950771 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:10.951478 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:10.951504 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:10.953991 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:10.957890 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:10.957898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:10.957912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:10.958423 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:10.958834 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:10.958837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:10.958840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:10.959014 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:10.959647 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:10.959653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:10.960041 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 10:07:10.960279 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 10:07:10.960678 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.162.60@KRBTEST.COM" created.
I20260924 10:07:10.963917 4381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:10 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:07:10.978696 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.964777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:43342)
0924 10:07:10.964966 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.964986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:10.965086 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:10.965995 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:10.965999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:10.966468 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:10.966638 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:10.966648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.968064 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.968069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:10.969194 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:10.969210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.970116 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.970127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:10.970344 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:10.970990 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:10.971007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:10.973178 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:10.976267 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:10.976272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:10.976277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:10.976629 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:10.976979 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:10.976984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:10.976988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:10.977131 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:10.977720 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:10.977725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:10.977882 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 10:07:10.977998 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:07:10.978276 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:07:10.978773 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.964903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.13.162.61:34601)
0924 10:07:10.965567 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:10.965582 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:10.965607 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:07:10.965651 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:10.965657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:10.965841 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:10.965959 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:10.966808 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.967928 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.969362 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:10.969969 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:10.970067 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:10.973384 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:10.973397 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:10.973412 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:10.973440 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:10.976141 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:10.976751 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:10.976758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:10.976760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:10.976880 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:10.977226 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:10.977230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:10.977232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:10.977599 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:10.977787 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:10.978148 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 10:07:10.978423 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":517,"thread_start_us":379,"threads_started":1}
I20260924 10:07:10.981423 4381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:10.982565 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.982086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58148)
0924 10:07:10.982235 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.982408 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 10:07:10.982792 4330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:10.984277 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:10.983930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58154)
0924 10:07:10.984091 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:10.984171 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:07:10.984475 4381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.162.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab.
Entry for principal kudu/127.13.162.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab.
I20260924 10:07:10.991796 13960 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
/tmp/dist-test-taskYd9YqT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:38233
--webserver_interface=127.13.162.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab
--principal=kudu/127.13.162.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.162.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.162.62:37453,127.13.162.61:34601,127.13.162.60:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:07:11.033771 4381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 1
I20260924 10:07:11.034857 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.034425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58162)
0924 10:07:11.034604 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.034710 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:07:11.035862 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.035538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58174)
0924 10:07:11.035674 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.035751 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:07:11.036057 4381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:11.104283 4381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 2
I20260924 10:07:11.105438 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.104991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58184)
0924 10:07:11.105182 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.105285 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 10:07:11.106411 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.106089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58188)
0924 10:07:11.106197 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.106260 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 10:07:11.106607 4381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:11.130290 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } attempt: 5
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:07:11.144580 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.131113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.13.162.61:34601)
0924 10:07:11.131285 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.131290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.131430 (+ 140us) server_negotiation.cc:407] Connection header received
0924 10:07:11.131625 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.131630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.131696 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.131781 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.132789 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.133475 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.135220 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.135670 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.135763 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.139356 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.139361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.139364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.139401 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.141946 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.142679 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.142683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.142686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.142814 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.143190 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.143193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.143195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.143499 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.143635 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.144047 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.144275 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:07:11.144563 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.131035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:43346)
0924 10:07:11.131326 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.131348 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:11.131492 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:11.131842 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:11.131846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:11.132373 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:11.132644 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.132657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.133621 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.133626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.135067 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.135086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.135824 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.135830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.136071 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.136696 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:11.136718 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:11.139139 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:11.142102 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.142109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.142113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.142555 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.142932 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.142937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.142941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.143077 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.143615 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:11.143620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:11.143776 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 10:07:11.143899 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:07:11.144200 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 10:07:11.145551 4307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:11.146632 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.146196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58202)
0924 10:07:11.146402 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.146480 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:07:11.147507 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.147183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58206)
0924 10:07:11.147320 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.147410 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:07:11.147693 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:11.197820 4381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 3
I20260924 10:07:11.198905 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 1
I20260924 10:07:11.199069 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.198562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58208)
0924 10:07:11.198734 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.198873 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 10:07:11.199990 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.199466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58216)
0924 10:07:11.199677 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.199789 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:07:11.200273 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.199749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58224)
0924 10:07:11.200021 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.200137 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260924 10:07:11.200563 4381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:11.200963 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.200650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58232)
0924 10:07:11.200779 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.200857 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 10:07:11.201185 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:11.269441 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 2
I20260924 10:07:11.270622 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.270123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58240)
0924 10:07:11.270359 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.270465 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:07:11.271675 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.271348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58250)
0924 10:07:11.271492 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.271572 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:07:11.271885 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
W20260924 10:07:11.319842 4388 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:07:11.320214 4388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:11.320317 4388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:11.320418 4388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:11.327725 4388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:11.327834 4388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:11.327909 4388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:11.327978 4388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:11.328063 4388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:11.338426 4388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.162.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.162.62:37453,127.13.162.61:34601,127.13.162.60:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/krb5kdc/kudu_127.13.162.60.keytab
--principal=kudu/127.13.162.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.162.60:38233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.162.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:11.339775 4388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:11.341180 4388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:07:11.343778 4381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 4
I20260924 10:07:11.344934 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.344461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58254)
0924 10:07:11.344682 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.344772 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:07:11.345856 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.345542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58260)
0924 10:07:11.345682 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.345751 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:07:11.346041 4381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
W20260924 10:07:11.352972 4394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:11.353047 4396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:11.353436 4393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:11.353701 4388 server_base.cc:1034] running on GCE node
I20260924 10:07:11.354555 4388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:11.355998 4388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:11.357218 4388 hybrid_clock.cc:648] HybridClock initialized: now 1790244431357145 us; error 46 us; skew 500 ppm
I20260924 10:07:11.357542 4388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244431, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:11.360636 4388 init.cc:377] Logged in from keytab as kudu/127.13.162.60@KRBTEST.COM (short username kudu)
I20260924 10:07:11.362066 4388 webserver.cc:460] Webserver started at http://127.13.162.60:37587/ using document root <none> and password file <none>
I20260924 10:07:11.362783 4388 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:11.362884 4388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:11.363161 4388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:11.366391 4388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data/instance:
uuid: "1ea7aacab76547b79652a622794bd5a3"
format_stamp: "Formatted at 2026-09-24 10:07:11 on dist-test-slave-jvsj"
server_key: "7a1e68244e04e36f3cb59195ab175948"
server_key_iv: "dcb8e5238d50be3e83feb115967dc69c"
server_key_version: "encryptionkey@0"
I20260924 10:07:11.367214 4388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal/instance:
uuid: "1ea7aacab76547b79652a622794bd5a3"
format_stamp: "Formatted at 2026-09-24 10:07:11 on dist-test-slave-jvsj"
server_key: "7a1e68244e04e36f3cb59195ab175948"
server_key_iv: "dcb8e5238d50be3e83feb115967dc69c"
server_key_version: "encryptionkey@0"
I20260924 10:07:11.372539 4388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:11.376313 4403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:11.377776 4388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:07:11.377966 4388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data,/tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal
uuid: "1ea7aacab76547b79652a622794bd5a3"
format_stamp: "Formatted at 2026-09-24 10:07:11 on dist-test-slave-jvsj"
server_key: "7a1e68244e04e36f3cb59195ab175948"
server_key_iv: "dcb8e5238d50be3e83feb115967dc69c"
server_key_version: "encryptionkey@0"
I20260924 10:07:11.378074 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 3
I20260924 10:07:11.378214 4388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:11.379115 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.378658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58264)
0924 10:07:11.378875 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.378962 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:07:11.380100 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.379730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58276)
0924 10:07:11.379923 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.379995 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 10:07:11.380280 4307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.162.60:38233: Network error: Client connection negotiation failed: client connection to 127.13.162.60:38233: connect: Connection refused (error 111)
I20260924 10:07:11.408545 4388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:11.412715 4388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:11.412992 4388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:11.444948 4388 rpc_server.cc:307] RPC server started. Bound to: 127.13.162.60:38233
I20260924 10:07:11.445004 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.162.60:38233 every 8 connection(s)
I20260924 10:07:11.446393 4388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/data/info.pb
I20260924 10:07:11.450271 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:11.455513 13960 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu as pid 4388
I20260924 10:07:11.455700 13960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYd9YqT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244085964582-13960-0/minicluster-data/master-2/wal/instance
I20260924 10:07:11.456225 13960 external_mini_cluster.cc:1428] Setting key 5034420e642ec945169fbbbf813d7362
I20260924 10:07:11.457646 4455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244431, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
I20260924 10:07:11.474962 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.460531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.13.162.62:37453)
0924 10:07:11.460739 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.460746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.461485 (+ 739us) server_negotiation.cc:407] Connection header received
0924 10:07:11.461723 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.461728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.461801 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.461881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.463143 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.463853 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.465327 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.465747 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.465866 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.469380 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.469385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.469389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.469422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.471913 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.472703 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.472708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.472710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.472849 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.473245 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.473249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.473251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.473507 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.473652 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.474373 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.474601 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:07:11.476042 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.458074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.13.162.60:38233)
0924 10:07:11.458930 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.458946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.458989 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:07:11.460796 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.460836 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.461315 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.461645 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.463093 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.464239 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.465509 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.466046 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.466168 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.469956 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.469967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.469982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.470018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.472812 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.473451 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.473460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.473463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.473585 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.473924 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.473929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.473933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.474283 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.474554 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.475042 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.475272 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"spinlock_wait_cycles":18688,"thread_start_us":379,"threads_started":1}
I20260924 10:07:11.476076 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.460424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.62:37453 (local address 127.0.0.1:53868)
0924 10:07:11.461323 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.461363 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:11.461549 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:11.461979 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:11.461986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:11.462629 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:11.462982 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.462999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.464003 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.464007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.465176 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.465188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.465982 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.465989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.466165 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.466876 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:11.466907 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:11.469185 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:11.472046 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.472058 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.472071 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.472584 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.472956 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.472961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.472965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.473141 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.473638 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:11.473646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:11.473978 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 10:07:11.474238 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 10:07:11.474780 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":5,"thread_start_us":492,"threads_started":1}
I20260924 10:07:11.477602 4455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244431, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.62@KRBTEST.COM
I20260924 10:07:11.491221 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.478406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.61:34601 (local address 127.0.0.1:43362)
0924 10:07:11.478644 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.478658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:11.478759 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:11.479104 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:11.479109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:11.479571 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:11.479747 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.479759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.480721 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.480726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.481723 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.481737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.482434 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.482440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.482659 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.483240 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:11.483256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:11.485436 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:11.488703 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.488709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.488713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.489217 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.489579 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.489584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.489588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.489731 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.490258 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:11.490262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:11.490442 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 10:07:11.490567 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 10:07:11.490868 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260924 10:07:11.491254 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.478380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.13.162.61:34601)
0924 10:07:11.478551 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.478556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.478735 (+ 179us) server_negotiation.cc:407] Connection header received
0924 10:07:11.478883 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.478888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.478968 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.479064 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.479893 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.480583 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.481852 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.482254 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.482419 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.485613 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.485618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.485621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.485653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.488533 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.489316 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.489330 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.489334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.489482 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.489831 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.489837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.489841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.490144 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.490277 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.490677 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.490880 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:07:11.492520 4455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:07:11.493043 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.478822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.13.162.62:37453)
0924 10:07:11.478980 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.478986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.479411 (+ 425us) server_negotiation.cc:407] Connection header received
0924 10:07:11.479554 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.479558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.479614 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.479685 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.480860 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.481582 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.483048 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.483527 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.483628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.487549 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.487557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.487563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.487602 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.490532 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.491135 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.491141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.491145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.491283 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.491675 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.491680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.491684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.491986 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.492228 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.492550 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.492763 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244431, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.60@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.162.61@KRBTEST.COM
I20260924 10:07:11.506814 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.493394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.13.162.60:38233)
0924 10:07:11.493896 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.493900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.493914 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:07:11.493949 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.493952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.494019 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.494109 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.494942 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.495973 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.497428 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.497971 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.498095 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.501520 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.501526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.501532 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.501571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.504156 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.504928 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.504933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.504937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.505092 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.505483 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.505486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.505488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.505780 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.505923 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.506383 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.506577 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:07:11.507156 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.493203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58292)
0924 10:07:11.493367 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.493385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:11.493481 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:11.494157 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:11.494160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:11.494615 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:11.494799 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.494811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.496096 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.496099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.497294 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.497308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.498119 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.498123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.498341 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.498885 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:11.498901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:11.501316 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:11.504282 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.504287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.504291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.504818 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.505208 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.505213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.505216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.505375 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.505899 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:11.505903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:11.506085 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 10:07:11.506254 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 10:07:11.506756 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":130}
I20260924 10:07:11.510293 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.494629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.13.162.61:34601)
0924 10:07:11.494799 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.494805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.494881 (+ 76us) server_negotiation.cc:407] Connection header received
0924 10:07:11.495038 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.495042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.495097 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.495164 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.496555 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.497533 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.498985 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.499601 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.499724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.504141 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.504148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.504151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.504194 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.507487 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.508332 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.508338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.508341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.508519 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.508929 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.508933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.508935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.509217 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.509428 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.509796 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.510023 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:07:11.512935 4455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: Bootstrap starting.
I20260924 10:07:11.515031 13960 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:11.516105 4455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:11.517151 4455 log.cc:824] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:11.520401 4455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: No bootstrap required, opened a new log
I20260924 10:07:11.517395 4282 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "905c660cd9da4df186ac5c4d6f691bfa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53874
I20260924 10:07:11.531080 4455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.531463 4455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7aacab76547b79652a622794bd5a3, State: Initialized, Role: FOLLOWER
I20260924 10:07:11.531949 4455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.532905 4461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.533274 4461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.533931 4455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:07:11.543416 4472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:07:11.543573 4472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:07:11.541334 4388 master_runner.cc:186] Error getting master registration for 127.13.162.62:37453: OK, instance_id { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" instance_seqno: 1790244430418437 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:07:11.549695 4388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:07:11.557479 4307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 4
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.62@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
I20260924 10:07:11.574126 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.558175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58296)
0924 10:07:11.558450 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.558472 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:11.558586 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:11.558946 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:11.558952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:11.559461 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:11.559667 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.559678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.561138 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.561144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.562632 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.562648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.563665 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.563670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.563880 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.564553 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:11.564575 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:11.567218 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:11.571284 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.571288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.571291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.571811 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.572216 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.572220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.572224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.572380 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.573111 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:11.573116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:11.573286 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 10:07:11.573419 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:07:11.573703 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 10:07:11.574302 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.558375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.13.162.60:38233)
0924 10:07:11.558605 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.558610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.558634 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:07:11.558703 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.558708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.558786 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.558912 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.559857 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.560999 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.562800 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.563524 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.563653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.567413 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.567420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.567425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.567464 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.571141 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.571950 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.571955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.571959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.572108 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.572501 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.572506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.572510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.572994 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.573167 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.573567 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.573810 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:07:11.580292 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa: Bootstrap starting.
I20260924 10:07:11.583699 4307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:11.584719 4307 log.cc:824] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:11.587570 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa: No bootstrap required, opened a new log
I20260924 10:07:11.593098 4307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.593428 4307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905c660cd9da4df186ac5c4d6f691bfa, State: Initialized, Role: FOLLOWER
I20260924 10:07:11.593870 4307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.594820 4475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.595168 4475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.596000 4307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:11.596164 4282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:07:11.596429 4282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:11.600056 4282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.602617 4282 leader_election.cc:288] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [CANDIDATE]: Term 1 election: Requested vote from peers 9bc3d0ff39f44bb2a732b8801f98851b (127.13.162.61:34601), 1ea7aacab76547b79652a622794bd5a3 (127.13.162.60:38233)
I20260924 10:07:11.602929 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "905c660cd9da4df186ac5c4d6f691bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ea7aacab76547b79652a622794bd5a3"
I20260924 10:07:11.603451 4430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:11.606968 4430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905c660cd9da4df186ac5c4d6f691bfa in term 1.
I20260924 10:07:11.607661 4256 leader_election.cc:302] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7aacab76547b79652a622794bd5a3, 905c660cd9da4df186ac5c4d6f691bfa; no voters:
I20260924 10:07:11.608193 4475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:11.618345 4475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [term 1 LEADER]: Becoming Leader. State: Replica: 905c660cd9da4df186ac5c4d6f691bfa, State: Running, Role: LEADER
I20260924 10:07:11.619043 4475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.624127 4483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905c660cd9da4df186ac5c4d6f691bfa. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.624611 4483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:11.629428 4486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:11.645234 4381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } attempt: 5
I20260924 10:07:11.646010 4486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 10:07:11 dist-test-slave-jvsj krb5kdc[4227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244430, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.162.61@KRBTEST.COM for kudu/127.13.162.60@KRBTEST.COM
W20260924 10:07:11.657455 4261 tablet.cc:2369] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:11.663386 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.645948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.162.60:38233 (local address 127.0.0.1:58308)
0924 10:07:11.646457 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:11.646479 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:11.646612 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:11.647532 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:11.647537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:11.648050 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:11.648268 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.648283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.649737 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.649742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.651181 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:11.651198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.652308 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.652314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:11.652552 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.653241 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:11.653264 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:11.656317 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:11.660242 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.660248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.660253 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.660894 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.661391 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:11.661397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:11.661400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:11.661569 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:11.662289 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:11.662294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:11.662487 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 10:07:11.662648 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 10:07:11.662964 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339}
I20260924 10:07:11.664464 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.646863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.13.162.60:38233)
0924 10:07:11.647044 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.647049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.647069 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:07:11.647133 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.647138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.647215 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.647314 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.648489 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.649562 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.651373 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.651979 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.652100 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.656589 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.656598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.656603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.656652 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.660058 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.661060 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.661066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.661070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.661235 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.661708 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.661713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.661717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.662070 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.662222 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.663812 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.664111 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 10:07:11.665202 4409 tablet.cc:2369] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:11.670423 4381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: Bootstrap starting.
I20260924 10:07:11.674572 4381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:11.675923 4381 log.cc:824] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:11.686713 4430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [term 1 FOLLOWER]: Refusing update from remote peer 905c660cd9da4df186ac5c4d6f691bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:11.688485 4483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:11.707142 4498 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:07:11.708194 4461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905c660cd9da4df186ac5c4d6f691bfa. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.708528 4461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.714649 4240 master_runner.cc:186] Error getting master registration for 127.13.162.61:34601: OK, instance_id { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" instance_seqno: 1790244430921695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:07:11.726377 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.679279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.13.162.61:34601)
0924 10:07:11.679667 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.679673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.679788 (+ 115us) server_negotiation.cc:407] Connection header received
0924 10:07:11.680083 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.680089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.680169 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.680249 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.682691 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.683794 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.694428 (+ 10634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.695195 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.695311 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.711092 (+ 15781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.711102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.711108 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.711162 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.715304 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.721894 (+ 6590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.721901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.721906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.722144 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.723487 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.723494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.723498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.723983 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.724261 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.725710 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.725983 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260924 10:07:11.728787 4381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: No bootstrap required, opened a new log
I20260924 10:07:11.744731 4240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:07:11.729028 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.679491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.13.162.62:37453)
0924 10:07:11.679805 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.679810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.680763 (+ 953us) server_negotiation.cc:407] Connection header received
0924 10:07:11.681132 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.681138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.681218 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.681299 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.689123 (+ 7824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.690360 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.697049 (+ 6689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.701501 (+ 4452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.701636 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.716437 (+ 14801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.716446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.716452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.716505 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.720696 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.722781 (+ 2085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.722789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.722793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.723005 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.724787 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.724794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.724798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.725216 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.725443 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.728360 (+ 2917us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.728633 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260924 10:07:11.741382 4492 mvcc.cc:205] Tried to move back new op lower bound from 7332841192150233088 to 7332841191918022656. Current Snapshot: MvccSnapshot[applied={T|T < 7332841192150233088}]
I20260924 10:07:11.746176 4475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.746526 4475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:11.748386 4381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.748793 4500 mvcc.cc:205] Tried to move back new op lower bound from 7332841192150233088 to 7332841191918022656. Current Snapshot: MvccSnapshot[applied={T|T < 7332841192150233088}]
I20260924 10:07:11.748947 4381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc3d0ff39f44bb2a732b8801f98851b, State: Initialized, Role: FOLLOWER
I20260924 10:07:11.749536 4381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } }
I20260924 10:07:11.750953 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:11.680188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.13.162.60:38233)
0924 10:07:11.680358 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.680364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.687262 (+ 6898us) server_negotiation.cc:407] Connection header received
0924 10:07:11.687348 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.687354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.687464 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.687564 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.729999 (+ 42435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.731174 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.734707 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:11.735409 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:11.735533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:11.741311 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:11.741318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:11.741322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:11.741377 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:11.745547 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.747280 (+ 1733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.747286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.747289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.747472 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:11.748161 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:11.748166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:11.748169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:11.748566 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:11.748859 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:11.749879 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 10:07:11.750272 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:07:11.755110 4486 catalog_manager.cc:1347] Generated new cluster ID: 0c464db257034fada162f270b6cfd754
I20260924 10:07:11.755226 4486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:11.755271 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.756160 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.757085 4381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:11.757167 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "905c660cd9da4df186ac5c4d6f691bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9bc3d0ff39f44bb2a732b8801f98851b"
I20260924 10:07:11.757668 4356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:11.761404 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905c660cd9da4df186ac5c4d6f691bfa in term 1.
I20260924 10:07:11.762547 4461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.762732 4461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.763124 4355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [term 1 FOLLOWER]: Refusing update from remote peer 905c660cd9da4df186ac5c4d6f691bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:11.764142 4475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:11.781713 4486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:11.783716 4486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:11.806242 4486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa: Generated new TSK 0
I20260924 10:07:11.807377 4486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:07:11.815451 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.815899 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.816197 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905c660cd9da4df186ac5c4d6f691bfa. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.816493 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:07:11.817991 4483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "905c660cd9da4df186ac5c4d6f691bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905c660cd9da4df186ac5c4d6f691bfa" member_type: VOTER last_known_addr { host: "127.13.162.62" port: 37453 } } peers { permanent_uuid: "9bc3d0ff39f44bb2a732b8801f98851b" member_type: VOTER last_known_addr { host: "127.13.162.61" port: 34601 } } peers { permanent_uuid: "1ea7aacab76547b79652a622794bd5a3" member_type: VOTER last_known_addr { host: "127.13.162.60" port: 38233 } } }
I20260924 10:07:11.818252 4483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905c660cd9da4df186ac5c4d6f691bfa [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:11.857852 4519 catalog_manager.cc:1259] Loaded cluster ID: 0c464db257034fada162f270b6cfd754
I20260924 10:07:11.857950 4519 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: loading cluster ID for follower catalog manager: success
I20260924 10:07:11.860754 4519 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: acquiring CA information for follower catalog manager: success
I20260924 10:07:11.873673 4519 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9bc3d0ff39f44bb2a732b8801f98851b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:07:11.974347 4310 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:11.960076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.13.162.62:37453)
0924 10:07:11.960296 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.960302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.972290 (+ 11988us) server_negotiation.cc:407] Connection header received
0924 10:07:11.972624 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.972632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.972733 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.972810 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.974088 (+ 1278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53882: BlockingRecv error: recv got EOF from 127.0.0.1:53882 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:07:11.982192 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:11.976353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.13.162.60:38233)
0924 10:07:11.976531 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.976537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.977125 (+ 588us) server_negotiation.cc:407] Connection header received
0924 10:07:11.977381 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.977387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.977489 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.977565 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.981942 (+ 4377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58320: BlockingRecv error: recv got EOF from 127.0.0.1:58320 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 10:07:11.983367 4385 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:11.966376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.13.162.61:34601)
0924 10:07:11.966599 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:11.966606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:11.975109 (+ 8503us) server_negotiation.cc:407] Connection header received
0924 10:07:11.981787 (+ 6678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:11.981793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:11.981899 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:11.981974 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:11.983144 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43400: BlockingRecv error: recv got EOF from 127.0.0.1:43400 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:07:12.043406 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.024283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.13.162.61:34601)
0924 10:07:12.024481 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.024487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.035287 (+ 10800us) server_negotiation.cc:407] Connection header received
0924 10:07:12.035919 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.035926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.036138 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.036299 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.037059 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.038214 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.040283 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.040959 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.041054 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.041844 (+ 790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.042591 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.042719 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.042954 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.043060 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 10:07:12.089073 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.076293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.13.162.60:38233)
0924 10:07:12.076484 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.076490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.086995 (+ 10505us) server_negotiation.cc:407] Connection header received
0924 10:07:12.087302 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.087310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.087516 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.087612 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.088790 (+ 1178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58336: BlockingRecv error: recv got EOF from 127.0.0.1:58336 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:07:12.141817 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.124091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.13.162.61:34601)
0924 10:07:12.124296 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.124304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.134955 (+ 10651us) server_negotiation.cc:407] Connection header received
0924 10:07:12.135504 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.135510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.135628 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.135694 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.136405 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.137624 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.139594 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.140232 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.140344 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.140943 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.141104 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.141165 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.141417 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.141502 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:07:12.163056 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.160896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.13.162.60:38233)
0924 10:07:12.161066 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.161072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.161563 (+ 491us) server_negotiation.cc:407] Connection header received
0924 10:07:12.161720 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.161726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.161829 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.161944 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.162827 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58348: BlockingRecv error: recv got EOF from 127.0.0.1:58348 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:07:12.176631 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.168472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.13.162.61:34601)
0924 10:07:12.168641 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.168648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.170293 (+ 1645us) server_negotiation.cc:407] Connection header received
0924 10:07:12.170575 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.170581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.170679 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.170781 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.171294 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.172439 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.174466 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.175109 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.175243 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.175737 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.175878 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.175969 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.176198 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.176290 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 10:07:12.185801 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.183787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.13.162.60:38233)
0924 10:07:12.184007 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.184014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.184129 (+ 115us) server_negotiation.cc:407] Connection header received
0924 10:07:12.184438 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.184444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.184558 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.184670 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.185543 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58350: BlockingRecv error: recv got EOF from 127.0.0.1:58350 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:07:12.200542 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.193567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.13.162.61:34601)
0924 10:07:12.193819 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.193826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.194362 (+ 536us) server_negotiation.cc:407] Connection header received
0924 10:07:12.194576 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.194582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.194673 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.194775 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.195231 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.196394 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.198451 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.199147 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.199281 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.199765 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.199907 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.200005 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.200159 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.200248 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260924 10:07:12.216572 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.208196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.13.162.60:38233)
0924 10:07:12.208485 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.208492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.214955 (+ 6463us) server_negotiation.cc:407] Connection header received
0924 10:07:12.215157 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.215163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.215297 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.215428 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.216332 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58356: BlockingRecv error: recv got EOF from 127.0.0.1:58356 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
I20260924 10:07:12.239880 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.232478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.13.162.61:34601)
0924 10:07:12.232680 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.232687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.233239 (+ 552us) server_negotiation.cc:407] Connection header received
0924 10:07:12.233598 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.233605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.233707 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.233814 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.234372 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.235637 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.237674 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.238378 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.238531 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.239031 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.239208 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.239315 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.239465 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.239557 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 10:07:12.254724 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.252428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.13.162.60:38233)
0924 10:07:12.252627 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.252634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.253102 (+ 468us) server_negotiation.cc:407] Connection header received
0924 10:07:12.253306 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.253312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.253409 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.253521 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.254463 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58368: BlockingRecv error: recv got EOF from 127.0.0.1:58368 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:07:12.282509 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.275230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.13.162.61:34601)
0924 10:07:12.275430 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.275437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.276004 (+ 567us) server_negotiation.cc:407] Connection header received
0924 10:07:12.276227 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.276233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.276343 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.276457 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.276957 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.278133 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.280117 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.280826 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.280975 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.281541 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.281692 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.281791 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.282021 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.282163 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 10:07:12.312598 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.300816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.13.162.60:38233)
0924 10:07:12.301016 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.301022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.311066 (+ 10044us) server_negotiation.cc:407] Connection header received
0924 10:07:12.311281 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.311287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.311399 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.311521 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.312355 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58382: BlockingRecv error: recv got EOF from 127.0.0.1:58382 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:07:12.354238 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.349222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.13.162.61:34601)
0924 10:07:12.349419 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.349425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.349992 (+ 567us) server_negotiation.cc:407] Connection header received
0924 10:07:12.350183 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.350187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.350260 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.350397 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.350785 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.351499 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.352788 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.353192 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.353299 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.353620 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.353717 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.353778 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.353878 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.353942 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:07:12.361088 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.359531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.13.162.60:38233)
0924 10:07:12.359747 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.359753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.359978 (+ 225us) server_negotiation.cc:407] Connection header received
0924 10:07:12.360118 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.360122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.360191 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.360269 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.360882 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58392: BlockingRecv error: recv got EOF from 127.0.0.1:58392 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:07:12.437865 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.432271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.13.162.61:34601)
0924 10:07:12.432489 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.432495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.432804 (+ 309us) server_negotiation.cc:407] Connection header received
0924 10:07:12.432983 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.432987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.433056 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.433131 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.433548 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.434385 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.436300 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.436719 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.436820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.437242 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.437342 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.437406 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.437526 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.437590 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:07:12.449376 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:12.447557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.13.162.60:38233)
0924 10:07:12.447742 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.447748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.448030 (+ 282us) server_negotiation.cc:407] Connection header received
0924 10:07:12.448199 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.448204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.448303 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.448397 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:12.449170 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58400: BlockingRecv error: recv got EOF from 127.0.0.1:58400 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:07:12.545567 4472 catalog_manager.cc:1259] Loaded cluster ID: 0c464db257034fada162f270b6cfd754
I20260924 10:07:12.545665 4472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: loading cluster ID for follower catalog manager: success
I20260924 10:07:12.548164 4472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: acquiring CA information for follower catalog manager: success
I20260924 10:07:12.549983 4472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ea7aacab76547b79652a622794bd5a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:07:12.585871 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.580786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.13.162.61:34601)
0924 10:07:12.581028 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.581035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.581353 (+ 318us) server_negotiation.cc:407] Connection header received
0924 10:07:12.581571 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.581575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.581647 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.581722 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.582163 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.582960 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.584303 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.584710 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.584828 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.585202 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.585315 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.585390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.585526 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.585609 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:07:12.596866 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.591624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.13.162.60:38233)
0924 10:07:12.591852 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.591858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.592071 (+ 213us) server_negotiation.cc:407] Connection header received
0924 10:07:12.592207 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.592211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.592365 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.592464 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.592751 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.593479 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.594807 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.595178 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.595267 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.595712 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.596254 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.596358 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.596489 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.596557 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:07:12.609223 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.602085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.13.162.61:34601)
0924 10:07:12.602440 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.602447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.603735 (+ 1288us) server_negotiation.cc:407] Connection header received
0924 10:07:12.604022 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.604027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.604110 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.604170 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.604942 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.605662 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.607322 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.607833 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.607945 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.608532 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.608633 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.608679 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.608881 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.608964 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260924 10:07:12.609812 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.603168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.13.162.60:38233)
0924 10:07:12.603349 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.603355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.603978 (+ 623us) server_negotiation.cc:407] Connection header received
0924 10:07:12.604212 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.604217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.604304 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.604377 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.604992 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.605929 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.607902 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.608319 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.608426 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.608933 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.609047 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.609136 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.609493 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.609570 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:07:12.610411 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:12.602078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.13.162.62:37453)
0924 10:07:12.602609 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:12.602616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:12.603279 (+ 663us) server_negotiation.cc:407] Connection header received
0924 10:07:12.603409 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:12.603414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:12.603677 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:12.603863 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:12.604412 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.605496 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.607673 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:12.608280 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:12.608409 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:12.608948 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:12.609344 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:12.609453 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:12.609853 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 10:07:12.610116 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":382,"threads_started":1}
I20260924 10:07:12.615095 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 4240
I20260924 10:07:12.616089 4302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:12.724575 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 4240
I20260924 10:07:12.741927 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 4314
I20260924 10:07:12.743018 4376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:12.849798 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 4314
I20260924 10:07:12.866214 13960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu pid 4388
I20260924 10:07:12.867321 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:12.974282 13960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYd9YqT/build/asan/bin/kudu with pid 4388
2026-09-24T10:07:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4277 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9036 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (346975 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST